AI-text detection
- Score
- 77.1
- Wallet spend
- $38.24 of $200
- Model cost
- $2.15
- Agent time
- 9h 17m
- Steps
- 930
- Tokens in / out
- 91.1M / 818k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 12th of 19
Score
77.1raw metric 0.7708
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.2292
- reward
- 0.7708
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9552
- threshold_5pct
- 0.9447
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.1367
- macro_auroc
- 0.9307
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.py loads one XLMRobertaForSequenceClassification (num_labels=1) from MODEL_DIR (default ./model) with HF_HUB_OFFLI
serve.py loads one XLMRobertaForSequenceClassification (num_labels=1) from MODEL_DIR (default ./model) with HF_HUB_OFFLINE=1. Each p_ai comes from that model's logit through a fixed monotone squash. There is no ensemble, no remote API and no second model in the serving path.
✓no retrieval of human textThe serving code makes no index, corpus or network lookups. Only model/ weights, the tokenizer files and the wheelhouse
The serving code makes no index, corpus or network lookups. Only model/ weights, the tokenizer files and the wheelhouse are used at inference, and no human-text corpus is loaded by serve.py.
✓stateless per request scoringThe worker batches queued requests only for throughput. Each item is tokenized with padding and an attention mask, and i
The worker batches queued requests only for throughput. Each item is tokenized with padding and an attention mask, and its logit is mapped independently. There is no cache, no cross-request statistics and no state that changes between requests.
✓no evaluation special casingThe /detect handler only pulls the text field and scores it. There are no hash or ID lookups, no text-pattern overrides
The /detect handler only pulls the text field and scores it. There are no hash or ID lookups, no text-pattern overrides and no references to evaluation data in serve.py or serve.sh.
✓own training data provenanceThe training data and its declared public sources are COLING2025 *_train.jsonl files, SemEval2024-task8 train parquet fi
The training data and its declared public sources are COLING2025 *_train.jsonl files, SemEval2024-task8 train parquet files (subtasks A, B and C), Wikipedia/Wikisource dumps, and C4 multilingual train shards. src/prep_corpus.py globs only '*_train.jsonl'. The download scripts fetch train files and train shards only. On top of that the candidate added its own generations (local Qwen/NLLB/mBERT models and spin-proxy API models) and the 200 allowed shipped examples. The proxy selection set is built from the candidate's own pools. Nothing references DetectRL-X, the forbidden frontier-ml-tasks repo or validation/test splits. COLING dev/test files were only listed during exploration, never downloaded or used. Download logs are retained in logs/download.log.
✓actual inference uses reviewed weightsserve.sh runs serve.py with MODEL_DIR defaulting to the shipped model/ directory, which holds model/model.safetensors an
serve.sh runs serve.py with MODEL_DIR defaulting to the shipped model/ directory, which holds model/model.safetensors and model/config.json. That is the only weight file in the inventory. src/train.py initializes from facebookai/xlm-roberta-large (the pinned revision is downloaded by src/download_all.sh) or from its own earlier checkpoint during refinement rounds, which matches architecture.json.
✓harness credentials used only by the harnessAPI generation used a spin OpenRouter proxy key created with `spin openrouter key create` (trajectory#581), stored in a
API generation used a spin OpenRouter proxy key created with `spin openrouter key create` (trajectory#581), stored in a key file and copied to the GPU host (trajectory#585). src/gen_api.py calls only https://api.spin.inductionlabs.com/v1. I found no use of the harness's ANTHROPIC/OPENAI keys or base URLs and no extra harness CLI launches.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed49m 36s into the runtook 7m 51s
startup 51.5 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 51.5 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.640196767392832}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.2 s for the first request, then 256.0 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed2h 48m into the runtook 9m 15s
startup 61.8 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 61.8 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.6401519454237405}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.4 s for the first request, then 311.1 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 3passed8h 36m into the runtook 8m 33s
startup 55.0 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 55.0 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.959413846039872}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.3 s for the first request, then 269.7 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 4passed9h 10m into the runtook 7m 1s
startup 45.5 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 45.5 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.959413846039872}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.3 s for the first request, then 301.2 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final state of /workspace/output (131 files, 6.22 GB, submitted 2026-10-21 04:49:38 UTC):
Model — facebookai/xlm-roberta-large @ c23d21b0620b635a76227c604d44e43a9f0ee389 (general multilingual encoder, not a detection model), full fine-tune with a single-logit head, fp32 weights. Shipped checkpoint = round 5 (2 epochs over 190,657 rows + a hard-example refinement pass).
Training data — 95,565 human / 95,092 machine rows, 8 languages, all 16 benchmark conditions, 8 generator families: M4/COLING train splits, Qwen2.5-7B, NLLB-200, bert-base-mul, and machine passages written through spin (gpt-4o-mini, gemini-2.5-flash, deepseek-chat, llama-3.3-70b). Human and machine sides are matched per language/domain/source/length so no shortcut survives; a language filter drops mislabelled rows. No public validation/test split was used for training, calibration or selection — the checkpoint was selected on a self-built proxy set (625 cells) and the 200 provided passages were used as extra training rows and as a sanity check.
Serving — serve.sh installs 40 vendored PyPI/PyTorch wheels into /tmp/aidet_pylibs, loads the model from model/ offline, and serves /health + /detect through a stdlib threaded HTTP server with one batching GPU worker (batch 48, max_len 512) and a strictly monotone non-saturating squash.
Verification — final frontier selfcheck 8313623d1f5e passed: startup 45.5 s of 300 s, 16.58 passages/s at 32 concurrent requests (requirement 10; 5,000 passages in 301 s of the 500 s allowed), all 5,000 responses parsed, server healthy afterwards. Three earlier checks also passed.
Caveats, stated plainly: the model API returned HTTP 409 from mid-run, so only 4 of 9 requested generator families (mistral, phi, claude, grok, kimi) delivered passages; the GPU lease expired at 04:35 UTC, so a round-7 rebuild aimed at the weakest condition (general_64, recall 0.68) was abandoned and its logs lost with the machine — both are documented in logs/round6_round7_notes.md. Proxy-set macro FNR at 1% FPR is 0.0918 (score 0.9082); the round-6 refinement that scored 0.9040 was discarded.