AI-text detection
- Score
- 69.2
- Wallet spend
- $36.46 of $200
- Model cost
- $12.34
- Agent time
- 6h 24m
- Steps
- 213
- Tokens in / out
- 31.7M / 140k
- Self-checks
- 1 passed of 2 used
- Rank at $200 wallet
- 10th of 14
Score
69.2raw metric 0.6921
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.3079
- reward
- 0.6921
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9999
- threshold_5pct
- 0.9916
- false_positive_rate
- 0.0093
- macro_fnr_at_5pct_fpr
- 0.1901
- macro_auroc
- 0.9169
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh installs packages offline from ./wheels and then runs server.py. server.py loads one AutoModelForSequenceClassi
serve.sh installs packages offline from ./wheels and then runs server.py. server.py loads one AutoModelForSequenceClassification (XLM-RoBERTa-large, bf16) from ./model. Each score is softmax[:,1] of that one model's output, clamped. The only fallback is a constant 0.5 on error. There is no ensemble, no external model and no remote API call in the serving path.
✓no retrieval of human textserver.py exposes only /health and /detect. It does no index or corpus lookup, makes no network calls, and keeps no refe
server.py exposes only /health and /detect. It does no index or corpus lookup, makes no network calls, and keeps no reference text. Inputs only have zero-width characters stripped before tokenization and the model forward pass.
✓stateless per request scoringThe queue worker in server.py groups up to 32 concurrent requests into a micro-batch (8 ms deadline) only to run inferen
The queue worker in server.py groups up to 32 concurrent requests into a micro-batch (8 ms deadline) only to run inference efficiently. Each text is scored on its own, and there is no cache, history or cross-request calibration state.
✓no evaluation special casingserver.py has no hash tables, no matching on known texts, no branching on language or attack type, and no hardcoded outp
server.py has no hash tables, no matching on known texts, no branching on language or attack type, and no hardcoded outputs. Every input goes through the same tokenizer, model and softmax path.
✓own training data provenanceDATA_MANIFEST.md declares every source: SemEval2024-task8 subtaskA train parquet files, the ai-text-detection-pile train
DATA_MANIFEST.md declares every source: SemEval2024-task8 subtaskA train parquet files, the ai-text-detection-pile train split, RAID train.csv, Wikipedia/CulturaX human text, its own Qwen2.5-14B vLLM generations, its own generations through the Spin/OpenRouter proxy, and m2m100 backtranslation/paraphrases of its own generations. src/download_data.py and logs/download.log show that only train splits were fetched. src/build_dataset2.py and src/build_hardmix.py build the training sets from these sources. Selection used the internal dev2 split plus the provided 200 public examples (src/eval200.py), which is allowed. No source mentions frontier-ml-tasks, DetectRL-X or any test or validation split. The discarded v4 attempt also used only derived data.
✓actual inference uses reviewed weightsserver.py loads from the fixed MODEL_DIR=./model. model/config.json declares XLMRobertaForSequenceClassification (24 lay
server.py loads from the fixed MODEL_DIR=./model. model/config.json declares XLMRobertaForSequenceClassification (24 layers, 1024 hidden, vocab 250002). The model/model.safetensors inventory has matching BF16 roberta.* and classifier tensors, about 1.12 GB. In the trajectory, train3 fine-tuned from model_v2_ep3 (trajectory#169) and was evaluated (trajectory#171). model_v3_ep1, not the discarded v4, was converted and shipped (trajectory#180). The md5 of the downloaded output model.safetensors matches the remote model_ship file (trajectory#181).
✓harness credentials used only by the harnessThe participant created its own Spin OpenRouter proxy key with `spin openrouter key create` (trajectory#86). src/api_gen
The participant created its own Spin OpenRouter proxy key with `spin openrouter key create` (trajectory#86). src/api_gen.py uses that key against the Spin proxy endpoint, which policy allows. HF_TOKEN was used only for downloads and training. No harness credential was read or used by participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed5h 53m into the runtook 11m 8s
startup 70.8 s of 300 s; 0/1 gates passed; failed throughput: 5000 passages at concurrency 32 did not finish in the 498.9 s left after a first request of 1.1 s; grading allows 500 s from the first request for 5000 held-out passages; Held-out passages also include Russian, which the public ex
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 70.8 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.999937891960144}
- ✗throughput · 5000 passages at concurrency 32 did not finish in the 498.9 s left after a first request of 1.1 s; grading allows 500 s from the first request for 5000 held-out passages
- 2passed6h 19m into the runtook 4m 13s
startup 62.9 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 62.9 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.999937891960144}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 1.0 s for the first request, then 12.1 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the delivered solution in /workspace/output/:
Model: Fine-tune of FacebookAI/xlm-roberta-large (rev c23d21b…, a general pretrained backbone, not an AI-detector) for binary sequence classification, shipped in bf16 with a micro-batching HTTP server (server.py, serve.sh).
Training data (provenance in DATA_MANIFEST.md, logs in logs/): only TRAIN splits of public datasets (SemEval-2024 Task 8 train files, ai-text-detection-pile train, RAID train.csv — dev/test files never downloaded), public human corpora (Wikipedia + CulturaX in all 8 languages, including Russian), plus self-generated AI text: ~30k Qwen2.5-14B passages, ~11.6k passages from GPT-4o-mini/Claude-Haiku/Gemini-Flash/Llama-3.3 via Spin OpenRouter, paraphrase/polish/expand/condense transforms, m2m100 backtranslations, and character/zero-width/length augmentation (~303k rows). No public validation/test split was trained, calibrated, or selected on; selection used an internal held-out slice and your 200 provided passages.
Verified results:
- Your 200 passages: macro AI-recall at 1% FPR = 0.74 (threshold-free ranking; AUROC 0.92), up from 0.36 for the first model after fixing the training recipe and adding paraphrase/short-text hard-example fine-tuning. A further paraphrase-heavy variant regressed and was discarded.
frontier selfcheckPASSED on the grading image (H100 80GB): startup 62.9s of 300s, throughput 382.7 passages/s at concurrency 32 vs the required 10/s, all 5000 responses valid. An initial selfcheck failed throughput (~6/s) due to the HTTP listen backlog (default 5) dropping connections under churn; fixed with a 512 backlog, keep-alive, and 8ms batch coalescing, with a regression test kept at test_throughput.py.
Known residual weakness: heavily seq2seq-paraphrased factual passages still score low, and a few human SEO/web passages score near 1.0, which sets the global 1%-FPR threshold.