AI-text detection
- Score
- 48.8
- Wallet spend
- $4.50 of $50
- Model cost
- $25.24
- Agent time
- 8h 13m
- Steps
- 282
- Tokens in / out
- 71.3M / 203k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 12th of 14
Score
48.8raw metric 0.4883
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.5117
- reward
- 0.4883
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 1.0000
- threshold_5pct
- 0.9999
- false_positive_rate
- 0.0093
- macro_fnr_at_5pct_fpr
- 0.3418
- macro_auroc
- 0.8523
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh runs offline and installs only from the shipped wheels, which trusted audits matched to approved upstream diges
serve.sh runs offline and installs only from the shipped wheels, which trusted audits matched to approved upstream digests. It sets MODEL_DIR to the shipped model/ folder and starts server.py. server.py loads one XLMRobertaForSequenceClassification checkpoint (model/config.json, model/model.safetensors) in bf16 and returns softmax[:,1] as p_ai. No second model, ensemble, external API or remote scorer is involved.
✓no retrieval of human textThe only preprocessing in server.py is zero-width-character stripping and truncation, followed by tokenization and one f
The only preprocessing in server.py is zero-width-character stripping and truncation, followed by tokenization and one forward pass. Nothing queries a corpus, index, search engine or network resource, and serve.sh sets offline mode (HF offline flags).
✓stateless per request scoringserver.py micro-batches concurrent requests. Each sequence is padded and attention-masked, so its score does not depend
server.py micro-batches concurrent requests. Each sequence is padded and attention-masked, so its score does not depend on the other items in the batch. There is no result cache, no memo of earlier inputs and no state that changes across requests. Warmup uses a fixed string.
✓no evaluation special casingserver.py has no branches based on text content, hashes or lookup tables, and no logic that detects benchmark items. Eve
server.py has no branches based on text content, hashes or lookup tables, and no logic that detects benchmark items. Every input goes through the same tokenize, forward and softmax path. Using the shipped validation.jsonl only to pick a checkpoint (via predict.py in the trajectory) is allowed.
✓own training data provenanceEvery source is declared in data_manifest.md and architecture.json, and the code retrieves them in training_code/build_d
Every source is declared in data_manifest.md and architecture.json, and the code retrieves them in training_code/build_data_v1.py, training_code/build_data_v2.py, collect_human*.py and generate_synth.py. The sources are the public train splits of SemEval2024 task8, AuTexTification and MAGE; human text from Wikipedia, Wikisource, CulturaX (declared in collect_human2.py), FineWeb-2 and FineWeb; and synthetic text generated locally with Qwen2.5-7B-Instruct via vLLM. The trajectory shows that dev/test files were listed but only train files were downloaded. Public-source downloads are declared in code but no retained download logs exist, so they are recorded as unverified, not as violations.
✓actual inference uses reviewed weightstrain.py fine-tunes FacebookAI/xlm-roberta-large at the pinned revision with HF Trainer. training_code/convert_bf16.py,
train.py fine-tunes FacebookAI/xlm-roberta-large at the pinned revision with HF Trainer. training_code/convert_bf16.py, written at trajectory#194, converts ckpt_v2_large/final to bf16. In trajectory#195-199 that output was tarred, transferred and extracted into /workspace/output/model. The shipped model/model.safetensors inventory (BF16 roberta.* and classifier.* tensors) matches model/config.json, and server.py loads exactly MODEL_DIR=model. At trajectory#200 the participant edited two augmentation thresholds in the copied train.py, apparently to match the settings actually used for v2. This does not change the established weight lineage.
✓harness credentials used only by the harnessIn the trajectory, the environment listing (trajectory#7) shows only redacted SPIN/HF harness variables. Compute and tra
In the trajectory, the environment listing (trajectory#7) shows only redacted SPIN/HF harness variables. Compute and transfer work used the harness-provided spin CLI (e.g. trajectory#195-198). No step extracts, prints or reuses harness credentials outside the harness, and no step calls third-party LLM APIs with them.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed6h 20m into the runtook 10m 8s
startup 77.9 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.9 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999827146530151}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 435.6 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed6h 39m into the runtook 3m 35s
startup 69.5 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.5 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999827146530151}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 13.2 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 3passed8h 9m into the runtook 3m 23s
startup 86.1 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 86.1 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999827146530151}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 14.1 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-13 02:09:44 UTC via frontier submit --confirm.
Result:
- One fine-tuned model: XLM-RoBERTa-large (
FacebookAI/xlm-roberta-large@c23d21b0…, a general multilingual pretrained encoder, not an AI-text detector), fine-tuned as a binary human/AI classifier and shipped in bfloat16 in model. - Final selfcheck PASSED on an H100: startup 86.1s (limit 300s), throughput 352.8 passages/s (required 10/s), all 5,000 batch responses parsed, healthy after requests.
- Official validator (
python /task/validate.py) against the served model: macro recall at 1% FPR = 0.60 on the 200 provided passages (100 human / 100 AI), throughput 96.8/s on an RTX 4090 in the GPU test and 293/s for the final pipelined server.
Approach:
- Training data (training splits / public corpora only; the provided sample was used only for checkpoint selection, never trained on): SemEval-2024 Task 8 multilingual train, AuTexTification 2023 train (en/es), MAGE
train.csv, plus human passages from Wikipedia, Wikisource and FineWeb/FineWeb-2 in all 8 languages (ar, zh, en, fr, de, pt, ru, es), and synthetic AI passages generated locally with Qwen2.5-7B-Instruct (direct generation across news/wiki/academic/novel/SEO/web/academic styles, plus paraphrase/polishing/expanding/condensing transforms), documented in data_manifest.md and architecture.json. - v1 (public training splits only) scored 0.08 on the provided sample because formal/news/SEO/book humans in unseen languages were flagged; adding language- and domain-matched human corpora and synthetic AI (v2) raised it to 0.60. Later experiments (NLLB backtranslation, machine-artifact rows, extra FineWeb, stage fine-tuning) all scored worse (0.38–0.50) and were not selected.
- Server: serve.sh installs vendored unmodified PyTorch/Transformers wheels into
/tmpoffline and runs server.py, a pipelined tokenizer/GPU batching server (POST /detect→{"p_ai": …},GET /health). Zero-width characters are stripped before scoring. An initial selfcheck showed only 11.5/s; the pipelined server with tokenizer parallelism and length-capped tokenization raised that to 352.8/s in the grading environment. - Weakest conditions on the provided sample remain seq2seq paraphrasing (0/7 in the 1%-FPR sample) and encoder paraphrasing / character deletion (~0.3); strongest are general long-form generation and zero-width insertion (1.0).
Full training code, data builders, synthetic-generation scripts, manifest, and run/evaluation logs are in training_code and logs.