AI-text detection
- Score
- 89.5
- Wallet spend
- $38.14 of $200
- Model cost
- $4.94
- Agent time
- 2h 7m
- Steps
- 158
- Tokens in / out
- 16.2M / 186k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 5th of 14
Score
89.5raw metric 0.8952
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.1048
- reward
- 0.8952
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.4147
- threshold_5pct
- 0.0001
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.0664
- macro_auroc
- 0.9798
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh starts server.py offline (HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE are set). server.py builds one Detector, and
serve.sh starts server.py offline (HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE are set). server.py builds one Detector, and detector.py loads a single local XLMRobertaForSequenceClassification from model/ with local_files_only. Scores come from the mean margin over at most 3 windows, passed through a sigmoid with temperature and bias from model/calibration.json. The export uses one checkpoint (model_count 1, --weights 1). I found no ensemble, no external API and no second model.
✓no retrieval of human textInference only normalizes text (NFKC plus a static confusables map in text_processing.py), tokenizes it and runs the cla
Inference only normalizes text (NFKC plus a static confusables map in text_processing.py), tokenizes it and runs the classifier. Nothing looks up a human-text corpus or index, and nothing is retrieved over the network while serving.
✓stateless per request scoringserver.py micro-batches up to 32 concurrent requests, but each request gets its own future and its own result. If a batc
server.py micro-batches up to 32 concurrent requests, but each request gets its own future and its own result. If a batch fails, it falls back to scoring items one by one. There is no cache, no cross-request state and no score normalization across a batch.
✓no evaluation special casingThe only routes are POST /detect and GET /health. No IDs, hashes or known evaluation texts are checked, and every input
The only routes are POST /detect and GET /health. No IDs, hashes or known evaluation texts are checked, and every input goes through the same normalize, window, model and calibration path.
✓own training data provenanceThe sources are declared in architecture.json and in training/all_corpora_manifest.json and training/final_training_mani
The sources are declared in architecture.json and in training/all_corpora_manifest.json and training/final_training_manifest.json. (1) DetectRL-X at a pinned revision, train split only: prepare_data.py drops rows where split != train, and the manifest says no public validation or test splits were used. (2) The 200 provided examples from /task/public/validation.jsonl, which are allowed. (3) Spin/OpenRouter rewrites of train-split parents. (4) mT5 XLSum generations from train-split parents. Every augmentation script asserts source_split == train. Selection and calibration use hash-grouped diagnostics taken only from the train split and kept disjoint from the training groups (verify_provenance.py). Download commands and logs are kept in logs/download_commands.txt and logs/download_data.log. The session shows no use of the frontier-ml-tasks repo: a multitude search returned 401 and nothing was obtained.
✓actual inference uses reviewed weightsThe trajectory shows the full chain. run2 selected checkpoint-6250 by train-diagnostic macro recall, export_model.py exp
The trajectory shows the full chain. run2 selected checkpoint-6250 by train-diagnostic macro recall, export_model.py exported it to final_model, and that output was copied into model/. The SHA256 9dfa0fd3… was checked against export_report.json (trajectory#139, #145, #147). The shipped model/model.safetensors is a BF16 XLM-R classifier whose tensors match config.json. serve.sh does not override MODEL_DIR, so server.py loads <root>/model. The self-check log confirms model_directory /submission/model (trajectory#155).
✓harness credentials used only by the harnessOnly environment variable names were listed (trajectory#6). A Codex auth file was listed but never used (trajectory#17).
Only environment variable names were listed (trajectory#6). A Codex auth file was listed but never used (trajectory#17). Model calls in the session went through a Spin-created OpenRouter key passed to generate_rewrites.py via --credentials, which is allowed (trajectory#36, #42). The participant code never reads ANTHROPIC or OPENAI keys or base URLs, and no extra agent CLI was launched. The session ends with the official frontier selfcheck and submit only (trajectory#151, #157).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 23m into the runtook 9m 45s
startup 50.3 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 50.3 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999008472688409}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 15.4 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed2h 4m into the runtook 3m 12s
startup 58.8 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 58.8 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9992318945653539}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 17.5 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with one trained multilingual detector, offline dependencies, and training/provenance evidence.
- Supports all eight requested languages.
serve.shprovidesPOST /detect→{"p_ai": ...}.- Final H100 self-check passed: 58.8-second startup, 284 passages/second, all 5,000 responses valid.
- Supplied-example recall reached 100% at 1% FPR; those examples were used in training, so this is a development result, not held-out accuracy.