AI-text detection
- Score
- 44.2
- Wallet spend
- $3.19 of $10
- Model cost
- $16.19
- Agent time
- 2h 10m
- Steps
- 329
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 16th of 19
Score
44.2raw metric 0.4417
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.5583
- reward
- 0.4417
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9108
- threshold_5pct
- 0.7756
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.2184
- macro_auroc
- 0.9513
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh installs the vendored wheels offline and starts a single uvicorn worker running server.py. At startup, server.p
serve.sh installs the vendored wheels offline and starts a single uvicorn worker running server.py. At startup, server.py builds one MultilingualAIDetector (model_def.py: an XLM-RoBERTa-large backbone, a feature MLP and heads) from model/ with pretrained=False. It then loads model/detector.safetensors with strict=True. Each /detect request is scored only by this model plus deterministic features from features.py. No other model, external LM or API is called at inference. mBERT and m2m100 were used only offline, for training-data augmentation.
✓no retrieval of human textNeither server.py, model_def.py nor features.py contains a corpus, index, nearest-neighbour lookup or network call. The
Neither server.py, model_def.py nor features.py contains a corpus, index, nearest-neighbour lookup or network call. The features are deterministic text statistics plus stopword-based language ID. The checkpoint holds only network parameters and calibration buffers (lang_shift, dom_shift, len_shift, global_bias and similar), not stored text.
✓stateless per request scoringFor each request, server.py calls prepare_single_input and then score_single_item with apply_calibration=True. There is
For each request, server.py calls prepare_single_input and then score_single_item with apply_calibration=True. There is no cache, no state that updates across requests and no batch-relative normalization. The calibration parameters are fixed buffers loaded from the checkpoint.
✓no evaluation special casingScoring is the same for every input. There is no hashing or matching of IDs or texts, no lookup of known evaluation item
Scoring is the same for every input. There is no hashing or matching of IDs or texts, no lookup of known evaluation items and no behaviour that depends on the request ID. Calibration uses features derived from the input text (language probabilities, domain softmax, length bucket), applied the same way to all inputs.
✓own training data provenancearchitecture.json and data_manifest.json declare the corpora. Human text comes from the train splits of c4, wikipedia 20
architecture.json and data_manifest.json declare the corpora. Human text comes from the train splits of c4, wikipedia 20231101 and opus_books, and AI text from the COLING_2025_MGT_multingual train shard. MBZUAI/Bactrian-X is declared in collect_public_train.py, which is shipped code. Synthetic data was generated through the Spin OpenRouter API, and paraphrase augmentation used mBERT and m2m100. Calibration and selection used local_eval_v2 (built from train-split source groups) and the 200-example public validation.jsonl, which is allowed. The trajectory shows dataset metadata exploration on HF, but downloads only of train splits. There is no use of the inductionlabs/frontier-ml-tasks repo, no DetectRL-X test rows and no public test split; searches for 'frontier-ml-tasks', 'DetectRL' and split='test' returned nothing. The final recalibration (trajectory#313) used only cache_le_v3 (local_eval_v2) and the public validation cache. The public sources are recorded in data_download.log and the code, though the retained downloads cannot be fully re-verified, so they count as declared but unverified.
✓actual inference uses reviewed weightsThe weight inventory for model/detector.safetensors matches model_def.py: XLM-R-large backbone tensors in BF16 (250002x1
The weight inventory for model/detector.safetensors matches model_def.py: XLM-R-large backbone tensors in BF16 (250002x1024 embeddings, 24 layers), plus attn_pool, text_proj, feat_mlp, classifier (690 input dims), classifier_skip, domain_head, cond_head, perturbation_boost and the calibration buffers. server.py loads exactly this file with strict=True. The trajectory shows the run3 WiSE-FT model being copied into /workspace/output/model (trajectory#300 area, steps 286-307). The recalibrated detector.safetensors was then downloaded into /workspace/output/model (trajectory#313, #316). The final self-check served this folder and passed (trajectory#327).
✓harness credentials used only by the harnessThe session used the read-only HF_TOKEN for HF downloads and dataset metadata, which is allowed. generate_paired_conditi
The session used the read-only HF_TOKEN for HF downloads and dataset metadata, which is allowed. generate_paired_conditions.py obtains its base_url and key from `spin openrouter key create` (see also trajectory#73), not from harness credentials. Spin compute and s5cmd used the brokered session credentials. Searches found no use of the agent's own Gemini, Anthropic or other API keys. The Gemini quota errors are harness system messages, not participant calls. The frontier selfcheck and submit commands are the harness's own tools.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 44m into the runtook 5m 3s
startup 74.8 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.8 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.8677412867546082}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.0 s for the first request, then 90.2 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed2h 5m into the runtook 3m 47s
startup 62.3 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 62.3 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9051966667175293}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.0 s for the first request, then 76.3 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests