AI-text detection
- Score
- 93.8
- Wallet spend
- $26.76 of $50
- Model cost
- $7.03
- Agent time
- 5h 28m
- Steps
- 321
- Tokens in / out
- 118.7M / 375k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
93.8raw metric 0.9375
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.0625
- reward
- 0.9375
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.7017
- threshold_5pct
- 0.0640
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.0413
- macro_auroc
- 0.9815
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh installs wheels offline, sets HF_HUB_OFFLINE=1 and DETECTOR_MODEL_DIR=$HERE/model, then runs server.py. server.
serve.sh installs wheels offline, sets HF_HUB_OFFLINE=1 and DETECTOR_MODEL_DIR=$HERE/model, then runs server.py. server.py loads one local model: AutoModel from model/ with local_files_only and bf16, plus the linear head in model/head.safetensors. It scores each text by mean-pooling hidden layer 36, applying the head and a sigmoid. It makes no external LM or API call and uses no ensemble or second model.
✓no retrieval of human textAt serving time no corpus, index or nearest-neighbour store is loaded or queried. server.py only tokenizes the normalize
At serving time no corpus, index or nearest-neighbour store is loaded or queried. server.py only tokenizes the normalized input (textops.py: NFKC normalization, zero-width removal, whitespace collapse) and runs the model. provenance/generated_ai_passages.jsonl.gz is a provenance record that the serving path never opens.
✓stateless per request scoringThe micro-batcher in server.py groups queued requests (up to 32 per batch, 10 ms wait), pads them and applies attention-
The micro-batcher in server.py groups queued requests (up to 32 per batch, 10 ms wait), pads them and applies attention-masked mean pooling, so each row's score depends only on its own text. There is no cache, memoization or cross-request state.
✓no evaluation special casingserver.py and textops.py contain no branches keyed on specific texts, hashes, request order or evaluation metadata. Ever
server.py and textops.py contain no branches keyed on specific texts, hashes, request order or evaluation metadata. Every input goes through the same normalize, truncate (8000 chars / 512 tokens), model and sigmoid path.
✓own training data provenanceAll sources are declared in architecture.json, provenance/data_manifest.json and provenance/download_commands.md. traini
All sources are declared in architecture.json, provenance/data_manifest.json and provenance/download_commands.md. training/fetch_human.py loads every source with split='train' (FineWeb, FineWeb-2, Wikipedia, Wikisource, arXiv abstracts) and only MAGE train.csv, with the manifest stating that MAGE valid/test and RAID test were not used. AI text was generated through spin OpenRouter (training/orclient.py, trajectory#19, trajectory#20). training/make_final_train.py adds the 200 shipped public validation examples, which the policy permits (trajectory#1). Searches of the evidence found no access to inductionlabs/frontier-ml-tasks or DetectRL-X. The public sources' download commands are declared but the downloads themselves are unverified.
✓actual inference uses reviewed weightsThe final run in logs/F4.log trained Qwen/Qwen3-4B-Base (revision pinned in architecture.json) with training/train.py. I
The final run in logs/F4.log trained Qwen/Qwen3-4B-Base (revision pinned in architecture.json) with training/train.py. It used 90,128 rows, ran 2,816 steps and saved runs/F4_4b_full/merged, the merged LoRA weights plus head.safetensors and detector.json. The trajectory shows the launch (trajectory#243) and the copy of that merged output into the output model directory (trajectory#304), with matching shard sizes. The shipped weights are model/model-0000{1,2}-of-00002.safetensors and model/head.safetensors, which have weight inventories. These are exactly what server.py loads through DETECTOR_MODEL_DIR set in serve.sh.
✓harness credentials used only by the harnessParticipant code used only the spin-created OpenRouter key (via SPIN_KEY_FILE/or_key.json in training/orclient.py, creat
Participant code used only the spin-created OpenRouter key (via SPIN_KEY_FILE/or_key.json in training/orclient.py, created at trajectory#19) and HF_TOKEN for downloads, both of which the policy allows. The trajectory shows no use of harness API keys or base URLs by participant code.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 3m into the runtook 7m 30s
startup 74.2 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 74.2 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999999952729468}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 176.6 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed5h 19m into the runtook 8m 31s
startup 73.1 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 73.1 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999999999853311}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 175.8 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output (12 GB) was graded as it stood at 10:52 UTC.
What's submitted
- Model: Qwen3-4B-Base (revision
906bfd4b…), LoRA-fine-tuned as a binary classifier. Inputs are NFKC-normalized with zero-width characters stripped and truncated to 512 tokens. Mean-pooled hidden states feed a linear head, andp_ai = sigmoid(logit). - Serving:
serve.shinstalls the vendored, unmodified torch 2.8.0 (cu128) and transformers 4.57.1 wheels offline into/tmp, then starts a batching server withPOST /detectandGET /health. - Provenance:
architecture.json, a data manifest, download commands, generation and training logs, and the generated AI passages.
Checks
frontier selfcheckpassed twice on the final files: startup 73 s (limit 300), throughput about 28 passages/s (limit 10), and all 5,000 responses parsed.- Held-out results from earlier runs, excluding the final model because it trained on validation: the 4B run scored 0.930 on the public validation set (1.7B: 0.83–0.84). On held-out generator families it scored 0.967.
- The final model separated 360 human and 121 AI passages from data it never saw, with AUROC 1.0. That's a small sanity check on one generator family, not a benchmark.
Choices you should know about
- Validation data in training: the final model includes the 200 validation passages, upweighted ×3. You allowed this, but it means the validation score for the final model is not an honest estimate.
- Weakest cells: very short passages (64 tokens) with character perturbations, and decoder-style paraphrases. Validation paraphrases are the least reliable part of the evidence, since only 6 passages were in that condition.
- Architecture review: it didn't run during self-check, so it's unknown how the reviewer will score the provenance. Everything it might look at is in
architecture.json,provenance/, andlogs/. - Spend: about $27 of the $50 budget went to GPU leases and generator calls.