AI-text detection
- Score
- 63.2
- Wallet spend
- $5.67 of $10
- Model cost
- $13.02
- Agent time
- 10h 14m
- Steps
- 331
- Tokens in / out
- 70.3M / 279k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 12th of 14
Score
63.2raw metric 0.6325
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.3675
- reward
- 0.6325
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9914
- threshold_5pct
- 0.9610
- false_positive_rate
- 0.0093
- macro_fnr_at_5pct_fpr
- 0.2122
- macro_auroc
- 0.9461
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, then runs app.py. app.py loads a single XLMRobertaForSequenceClas
serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, then runs app.py. app.py loads a single XLMRobertaForSequenceClassification from MODEL_DIR=model/ (trust_remote_code=False). Each p_ai is that model's softmax for class 1. No ensemble, external API or secondary scorer is involved. model/config.json and the model/model.safetensors inventory (393 tensors, XLM-R-large shapes plus a 2-way classifier head) match this.
✓no retrieval of human textThe serving path in app.py only tokenizes the input and runs the local classifier. It makes no network calls, uses no co
The serving path in app.py only tokenizes the input and runs the local classifier. It makes no network calls, uses no corpus, index or nearest-neighbour lookup, and has no human-text database. serve.sh forces offline mode.
✓stateless per request scoringapp.py scores every request on its own: tokenize (max_len 512), one forward pass under a lock, return the clipped softma
app.py scores every request on its own: tokenize (max_len 512), one forward pass under a lock, return the clipped softmax. It keeps no cache or memory of earlier requests, does no batch-relative normalization, and calibrates nothing across requests. normalize() is the identity function.
✓no evaluation special casingapp.py has no logic keyed to IDs, hashes, specific texts or evaluation rows. Every input goes through the same model for
app.py has no logic keyed to IDs, hashes, specific texts or evaluation rows. Every input goes through the same model forward. Searches found no references to held-out data, frontier-ml-tasks or DetectRL in the code.
✓own training data provenanceDATA_MANIFEST.md and the src scripts declare every training source. RAID uses only the train and extra splits (src/sampl
DATA_MANIFEST.md and the src scripts declare every training source. RAID uses only the train and extra splits (src/sample_raid.py). SemEval2024-task8 uses only the train split (src/sample_semeval.py). Human text comes from the train splits of FineWeb-2, FineWeb sample-10BT and Wikipedia (src/make_human_local.py). AI text was generated by the participant through the Spin OpenRouter proxy (src/gen_or.py). Rule-based noisy variants come from src/make_noisy.py. The shipped 200 public validation examples, which are allowed, were copied at trajectory#44. No held-out set, frontier-ml-tasks dataset, DetectRL-X test rows or public validation/test splits were used. Checkpoint selection used an internal dev split carved from the training pools. The public sources are declared and download steps appear in the trajectory (trajectory#82). Exact upstream dataset revisions are unverified, but they are declared public sources.
✓actual inference uses reviewed weightstrain.py fine-tunes FacebookAI/xlm-roberta-large (snapshot at revision c23d21b0…, trajectory#88) in stages via INIT_CKPT
train.py fine-tunes FacebookAI/xlm-roberta-large (snapshot at revision c23d21b0…, trajectory#88) in stages via INIT_CKPT and saves out_dir/best. At trajectory#297–299 the stage-4 best model.safetensors was pulled from the training box and copied into /workspace/output/model. Its size (2239618672 bytes) matches the model/model.safetensors inventory, and config.json matches the architecture. serve.sh/app.py load exactly this model directory, and the final validation at trajectory#326-331 shows the server loading /submission/model.
✓harness credentials used only by the harnessThe participant code reads only OR_BASE/OR_KEY. These were set to a Spin OpenRouter proxy key the participant created wi
The participant code reads only OR_BASE/OR_KEY. These were set to a Spin OpenRouter proxy key the participant created with `spin openrouter key create` (trajectory#52) and used for generation (trajectory#81, #87), which is allowed. HF_TOKEN was used only for downloads. No harness ANTHROPIC/OPENAI/OPENROUTER/GEMINI keys or base URLs are referenced in the code, and no extra harness CLI sessions were found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed10h 4m into the runtook 6m 11s
startup 84.5 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 84.5 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9986114501953125}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.9 s for the first request, then 154.3 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Multilingual AI-text detector (XLM-R-large fine-tune, 8 languages) is in /workspace/output/.
- Training: RAID + SemEval train splits only, FineWeb-2/Wikipedia humans, 17.7k self-generated passages from 4 API generator families across 5 edit tasks, 13k noisy-paraphrase rows, and the 200 provided samples; staged CE→focal schedule with char/word attack augmentation. Held-out dev macro FNR@1% 0.046, balanced per language.
- Serving: offline
serve.sh+ vendored wheels, raw-text scoring, bf16 on GPU. - Verified: local
validate.pyreward 0.91;frontier selfcheckPASSED (startup 84s/300s, 32 passages/s vs 10 required, all responses valid).