AI-text detection
- Score
- 74.1
- Wallet spend
- $28.23 of $200
- Model cost
- $9.46
- Agent time
- 5h 50m
- Steps
- 388
- Tokens in / out
- 162.8M / 439k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 9th of 14
Score
74.1raw metric 0.7412
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.2588
- reward
- 0.7412
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9973
- threshold_5pct
- 0.9646
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.1029
- macro_auroc
- 0.9791
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, then runs server.py. server.py loads only the local members l
serve.sh sets HF_HUB_OFFLINE=1 and TRANSFORMERS_OFFLINE=1, then runs server.py. server.py loads only the local members listed in model/ensemble.json (member_a, member_c, member_d). Each member is an AutoModel backbone plus a Linear head from head.pt. The server averages the member logits with equal weights and applies a sigmoid. The serving path makes no remote API or network calls. All members were trained by the participant with training/train_clf.py, so the fixed equal-weight average acts as one self-trained detector.
✓no retrieval of human textserver.py only normalises the text, tokenises it and runs a forward pass. No corpus, index, embedding store or lookup is
server.py only normalises the text, tokenises it and runs a forward pass. No corpus, index, embedding store or lookup is packaged or called at inference. The training corpora remain in the training pipeline and are not consulted when scoring.
✓stateless per request scoringThe micro-batcher in server.py (MAX_BATCH 64, 8 ms wait) right-pads inputs and uses masked mean pooling. Each p_ai there
The micro-batcher in server.py (MAX_BATCH 64, 8 ms wait) right-pads inputs and uses masked mean pooling. Each p_ai therefore depends only on its own text. The server keeps no cache, request history, cross-request ranking or batch-relative calibration.
✓no evaluation special casingserver.py has no branches on request IDs, no hash or lookup tables of known passages, and no rules aimed at the evaluati
server.py has no branches on request IDs, no hash or lookup tables of known passages, and no rules aimed at the evaluation set. Every request goes through the same normalisation (NFKC plus removal of Cf characters, then truncation) and the same model forward pass.
✓own training data provenancetraining/data_manifest.json and training/DATA_PROVENANCE.md declare the human sources with pinned revisions: Wikipedia 2
training/data_manifest.json and training/DATA_PROVENANCE.md declare the human sources with pinned revisions: Wikipedia 20231101, FineWeb-2, FineWeb sample-10BT and Project Gutenberg, all train or unsplit. The AI text was generated by the participant with local open-weight generators (gen_ai.py), the Spin OpenRouter API (gen_api.py, gen_api2.py) and Marian backtranslation (gen_mt.py). The v4 hard negatives came from the participant's own v3 human windows (mine_hard.py). Download and extraction logs are retained (prep_*.log, gen_*.log). The shipped 200 public validation examples were used only for evaluation, ensemble selection and one augmentation rule, which the policy allows. I found no use of DetectRL-X test rows, inductionlabs/frontier-ml-tasks, or any public validation or test split.
✓actual inference uses reviewed weightsserve.sh defaults MODEL_DIR to the shipped model/ directory. server.py loads backbone/ and head.pt for each member in mo
serve.sh defaults MODEL_DIR to the shipped model/ directory. server.py loads backbone/ and head.pt for each member in model/ensemble.json. The backbone configs match the declared bases: Qwen2 at hidden size 1536 for member_a, Qwen2 at 3584 for member_c, and Qwen3 at 4096 for member_d. The trajectory shows how the shipped weights got there. C4 and D4 were packaged from their training runs and transferred over Spin R2. The safetensors size was checked against the node copy (trajectory#368). They were then copied into output/model/member_c and member_d (trajectory#365, trajectory#369). The final layout was served and validated (trajectory#372, trajectory#373), and the final selfcheck passed on this state (trajectory#384).
✓harness credentials used only by the harnessThe API generation in training/gen_api.py uses the Spin OpenRouter key from /tmp/orkey.json, which the policy allows (tr
The API generation in training/gen_api.py uses the Spin OpenRouter key from /tmp/orkey.json, which the policy allows (trajectory#24, trajectory#25). HF_TOKEN was used only for downloads. upload_C4.log records a Spin R2 transfer, not an HF upload. I found no use or copying of ANTHROPIC_API_KEY, OPENAI_API_KEY or GEMINI_API_KEY, the harness OpenRouter key, or any harness *_BASE_URL. No extra harness CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 58m into the runtook 4m 11s
startup 69.1 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 69.1 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999999625503116}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 29.6 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 2passed2h 54m into the runtook 6m 25s
startup 56.8 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 56.8 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999999822758406}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 100.0 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 3passed3h 12m into the runtook 11m 23s
startup 77.3 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.3 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999999823796472}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 194.3 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
- 4passed5h 38m into the runtook 11m 24s
startup 77.2 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 77.2 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9999998876809553}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.1 s for the first request, then 196.0 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: /workspace/output was submitted at 11:15 UTC. It is a three-member ensemble that passed the final frontier selfcheck.
What's submitted
- Members: Qwen2.5-1.5B fully fine-tuned, Qwen2.5-7B with LoRA, and Qwen3-8B with LoRA. Each has a mean-pooled linear head. Logits are averaged with equal weights, then passed through a sigmoid.
- Serving:
serve.shinstalls vendored cp312 wheels into/tmp(torch 2.8.0+cu128, transformers 5.17.0, and their dependencies), then startsserver.py. The server normalizes text (NFKC, invisible characters removed) and answers/detectand/health. - Final selfcheck: passed on the exact submitted state. Startup took 77 s of the 300 s budget. Throughput was 25.5 passages/s against the required 10. All 5,000 responses parsed.
- Documentation:
architecture.jsonlists base models with commit revisions and all evidence files, which exist.training/contains the scripts, logs,data_manifest.json, andDATA_PROVENANCE.mdwith HF dataset revisions and download commands.
Scores (my own measurements)
- Holdout (6,858 passages, grouped by source document, separate from training): macro FNR at 1% FPR gives a score of 0.922 for the ensemble. The best single member scored 0.903.
- Validation (your 200 passages, run through the official
validate.pyon the node with the submitted layout): 0.86. This set is small and its 1% threshold rests on about one passage, so treat it as a sanity check, not a precise estimate.
Choices and deviations to be aware of
- Training data: only train splits and un-split public corpora (Wikipedia 20231101, FineWeb-2, FineWeb sample-10BT, Gutenberg) plus AI text I generated with open models, OpenRouter, and Marian backtranslation. No public validation or test split was used.
- Your validation sample: I used it for evaluation and to compare candidate ensembles. It also informed one augmentation: per-character spacing for Chinese encoder paraphrases. Your rules allowed this, and I've disclosed it in the provenance file.
- Hard-negative mining: I added 3,000 human passages that the v2 ensemble scored as most AI-like. This was driven by validation human scores. It reduced the validation human tail from 6 to 2 passages above 0.99, with no change on the holdout.
- Failed run: the first 7B run finished training but lost its weights to a full disk. I documented this in the provenance file.
- Not in the submission: v3 runs. Their validation tail was much heavier, so I dropped them.
- Architecture review: the selfcheck does not run it, so the documentation is the only input to that review.
Spend was about $28 of the $200 budget.