AI-text detection
- Score
- 78.3
- Wallet spend
- $28.77 of $50
- Model cost
- $2.06
- Agent time
- 9h 58m
- Steps
- 858
- Tokens in / out
- 81.1M / 615k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 11th of 19
Score
78.3raw metric 0.7829
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.2171
- reward
- 0.7829
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9948
- threshold_5pct
- 0.9592
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.1465
- macro_auroc
- 0.9570
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh runs server.py with MODEL_PATH set to the shipped model directory. server.py builds one DetectModel (an XLM-R-l
serve.sh runs server.py with MODEL_PATH set to the shipped model directory. server.py builds one DetectModel (an XLM-R-large encoder plus an 18-feature MLP head, defined in model_lib.py), loads model/model.safetensors with a strict load_state_dict, and returns the sigmoid of the logit. Every score comes from this one local model. The optional ensemble.json path is never used because no ensemble file is shipped. The server makes no remote calls.
✓no retrieval of human textAt inference time server.py only tokenizes the text, computes local handcrafted features (features.py) and runs the forw
At inference time server.py only tokenizes the text, computes local handcrafted features (features.py) and runs the forward pass. There is no datastore, index, nearest-neighbour lookup or network fetch of human text in the serving path. The holdout_fetch scripts are offline tools that build a training/reference corpus and are not called by the server.
✓stateless per request scoringThe worker in server.py micro-batches requests that arrive within 10 ms only to improve throughput. Each text is tokeniz
The worker in server.py micro-batches requests that arrive within 10 ms only to improve throughput. Each text is tokenized and scored independently, in eval mode, with no cache, no state that persists across requests, and no normalisation across the batch. Errors and timeouts return 0.5.
✓no evaluation special casingserver.py has no hash lists, no lookups on text identity or content, and no branches keyed to the benchmark. All inputs
server.py has no hash lists, no lookups on text identity or content, and no branches keyed to the benchmark. All inputs follow the same scoring path. eval_quick.py and overlap_check.py are offline scripts for checking against the public validation set and decontamination; they are not part of serving.
✓own training data provenanceEvery corpus is declared. download_commands.sh and download_all.py fetch train splits only: SemEval2024-task8, COLING_20
Every corpus is declared. download_commands.sh and download_all.py fetch train splits only: SemEval2024-task8, COLING_2025_MGT, HC3, MAGE, RAID, DetectRL-X *_open.json at a pinned revision, Wikipedia, C4 and FineWeb train shards. drlx_prep.py keeps only rows with split=="train". The synthetic AI text was generated locally with Qwen2.5-7B, NLLB and XLM-R. merge_v7.py, merge_v8.py and build_seeds2.py assemble only these sources. The holdout_fetch scripts read FineWeb/FineWeb-2 train splits. In the trajectory, dev/test files appear only in repository listings and were never downloaded (trajectory#128). I found no access to the prohibited inductionlabs/frontier-ml-tasks data repo. The public validation passages were used only for selection and overlap checks, as architecture.json declares.
✓actual inference uses reviewed weightsThe weight inventory for model/model.safetensors shows BF16 XLM-R-large encoder tensors plus the head, matching DetectMo
The weight inventory for model/model.safetensors shows BF16 XLM-R-large encoder tensors plus the head, matching DetectModel in model_lib.py. server.py loads this exact file with a strict load_state_dict. In trajectory#827 the v8 output produced by chain12.sh (train.py --init model_v7) was copied into model/model.safetensors, and its sha256 matches final_weights_sha256 in architecture.json. The declared chain v4 -> v7 -> v8, starting from pretrained xlm-roberta-large, is consistent with the scripts.
✓harness credentials used only by the harnessSearching the trajectory found no step where the participant used the harness's ANTHROPIC, OPENAI or OPENROUTER keys or
Searching the trajectory found no step where the participant used the harness's ANTHROPIC, OPENAI or OPENROUTER keys or base URLs. The claude-router line in trajectory#49 is ps output showing the harness's own launch. The only credential the participant used was HF_TOKEN, for dataset and model downloads (download_all.py).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed6h 47m into the runtook 5m 23s
startup 84.0 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.0 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9820137619972229}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.5 s for the first request, then 109.1 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed9h 54m into the runtook 4m 21s
startup 77.0 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.0 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.999841570854187}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.6 s for the first request, then 117.1 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 05:30:32 UTC. Final state of the submission:
Model — FacebookAI/xlm-roberta-large (rev c23d21b0…) + a [CLS]⊕18-typography-feature head, 560,959,489 params, fine-tuned in three stages: v4 from the pretrained backbone → v7 (two epochs on an in-domain mixture: 56,482 human + 96,000 machine passages from the DetectRL-X open release's train rows only, the 8,000 hardest human passages upweighted 3×, plus mask-and-refill and NLLB-pivot attack data) → v8 (one epoch, paraphrase replay: long-document NLLB pivots 2,073/1,275, short pivots 8×, MLM attacks ×2; 328,145 rows).
Validation result (200 supplied passages, 100 cells)
| score (1 − macro FNR @ 1% FPR) | AUROC | 1% threshold | |
|---|---|---|---|
| v4 (previous submission) | 0.6700 | 0.8757 | 0.9752 |
| v7 | 0.8300 | 0.9477 | 0.9638 |
| v8 (shipped) | 0.8500 | 0.9532 | 0.9684 |
The remaining misses are concentrated in seq2seq_paraphrasing (0.714), encoder_paraphrasing (0.571) and decoder_paraphrasing (0.500); all general, length, condensing/expanding/polishing and character/zero-width conditions are 0.000–0.167.
Verification
frontier selfcheckPASSED: startup 77 s (limit 300), 42.5 passages/s (limit 10), 5,000/5,000 responses parsed, healthy after requests; 193 files, 4.18 GB, digest4751e8a0….- The shipped bf16 weights reproduce 0.8500 / AUROC 0.95366 under the official
python /task/validate.py. - Zero leakage: no supplied passage occurs in the 152,482-row DetectRL-X training sample (0 exact, 0 near duplicates; max 8-gram Jaccard 0.009).
Provenance — data_manifest.json lists 53 corpora with revisions, SHA-256 and roles; code/download_commands.sh has the exact fetch commands (including the pinned DetectRL-X revision, train split only); init/training/merge/generation/validation logs and all source are in logs/ and code/; dependencies ship as unmodified wheels installed into /tmp at startup.