AI-text detection
- Score
- 87.0
- Wallet spend
- $13.22 of $200
- Model cost
- $16.50
- Agent time
- 2h 28m
- Steps
- 340
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 8th of 19
Score
87.0raw metric 0.8698
The score is the raw metric (1 − macro false-negative rate at 1% false positives), which already runs from 0 to 1, times 100.
Verifier details
- metric
- macro_fnr_at_1pct_fpr
- macro_fnr_at_1pct_fpr
- 0.1302
- reward
- 0.8698
- reward_definition
- 1 - mean language/family/condition FNR at global 1% FPR
- humans
- 1928
- ai
- 3072
- threshold_1pct
- 0.9875
- threshold_5pct
- 0.3725
- false_positive_rate
- 0.0099
- macro_fnr_at_5pct_fpr
- 0.0391
- macro_auroc
- 0.9839
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓single local model scoresserve.sh starts serve.py offline (HF_HUB_OFFLINE=1). serve.py loads one detector from shipped files: the Qwen3-1.7B-Base
serve.sh starts serve.py offline (HF_HUB_OFFLINE=1). serve.py loads one detector from shipped files: the Qwen3-1.7B-Base weights in model/base_weights, the LoRA adapter in model/lora_adapter (merged at load time) and the custom head in model/detector_head.pt (QwenAIDetector in model_arch.py). The token-surprise features come from the logits of that same fine-tuned backbone, so they belong to the one trained detector. No second model or external API is used to score requests.
✓no retrieval of human textThe serving path (serve.py, model_arch.py) loads no text corpus, embedding index, hash table or lookup structure. Scores
The serving path (serve.py, model_arch.py) loads no text corpus, embedding index, hash table or lookup structure. Scores come only from the network's forward pass plus fixed scalar calibration rules.
✓stateless per request scoringserve.py micro-batches requests, pads them and passes an attention mask, so each row is pooled and classified independen
serve.py micro-batches requests, pads them and passes an attention mask, so each row is pooled and classified independently. Nothing is cached, persisted or accumulated across requests, and the calibration rules use only features of the current passage.
✓no evaluation special casingpreprocess_passage builds its structural header from the passage text alone. The post-classifier rules (a short-text boo
preprocess_passage builds its structural header from the passage text alone. The post-classifier rules (a short-text boost, and a boost for long, low-likelihood passages) are generic thresholds on features computed from the text. Nothing matches IDs, specific texts or hashes, and nothing depends on request order or on detecting evaluation traffic.
✓own training data provenanceDATA_MANIFEST.json and architecture.json declare the train-split corpora with revisions, the AI generations made through
DATA_MANIFEST.json and architecture.json declare the train-split corpora with revisions, the AI generations made through the session's own Spin OpenRouter key, and the paraphrase/augmentation models (m2m100, mBERT). The public validation examples (/task/public/validation.jsonl) are used for model selection and stage-3 fine-tuning, which is allowed. symanto/autextification2023 (train split) is declared in src/collect_human_and_public_train.py but not in the manifest. It is a public train split, and its download log was not independently verified. The trajectory shows only README downloads plus a directory listing that included test/valid files of other datasets; nothing shows those test files were used. No forbidden source was found.
✓actual inference uses reviewed weightsserve.py loads exactly the shipped model/base_weights/model.safetensors, model/lora_adapter/adapter_model.safetensors an
serve.py loads exactly the shipped model/base_weights/model.safetensors, model/lora_adapter/adapter_model.safetensors and model/detector_head.pt, with no runtime-selectable alternative. The base model and revision match the training code. The trajectory shows these files copied from the stage-3 best_model checkpoint produced by the training scripts (trajectory#306), and the base weights downloaded at the pinned revision (trajectory steps 122-148).
✓harness credentials used only by the harnessThe OpenRouter key used for data generation (/workspace/work/or_key.json, read in src/generate_openrouter_ai.py) was the
The OpenRouter key used for data generation (/workspace/work/or_key.json, read in src/generate_openrouter_ai.py) was the participant's own, created with 'spin openrouter key create' (trajectory steps 71-87). The trajectory's S3 transfers use Spin's transfer credentials to move the participant's own checkpoint. Neither the code nor the trajectory shows use of harness-reserved API keys or endpoints.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 21m into the runtook 5m 30s
startup 64.2 s of 300 s; 1/1 gates passed; Held-out passages also include Russian, which the public examples lack, and are about 9% longer in UTF-8 bytes, so keep more than that to spare.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 64.2 s
- ✓POST /detect · {"id": "validation-000000", "p_ai": 0.9937066435813904}
- ✓throughput · 5000 public passages (the 200 examples resampled) at concurrency 32: 0.6 s for the first request, then 98.4 s for the batch; grading allows 500 s from the first request for 5000 held-out passages
- ✓batch responses · all 5000 responses parsed
- ✓healthy after requests