Zero-shot classification
- Score
- 17.4
- Wallet spend
- $14.19 of $200
- Model cost
- $8.30
- Agent time
- 43m
- Steps
- 138
- Tokens in / out
- 14.8M / 50k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 11th of 14
Score
17.4raw metric 0.4144
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.553 | 0.223 | 1.00 | 42.5 |
| amazonpolarity | 0.681 | 0.522 | 1.00 | 33.2 |
| appreviews | 0.761 | 0.521 | 1.00 | 50.2 |
| banking77 | 0.233 | 0.0142 | 1.00 | 22.2 |
| biasframes_intent | 0.484 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.520 | 0.456 | 1.00 | 11.7 |
| biasframes_sex | 0.355 | 0.369 | 1.00 | 0.0 |
| capsotu | 0.0889 | 0.0166 | 1.00 | 7.3 |
| emotiondair | 0.283 | 0.132 | 1.00 | 17.4 |
| empathetic | 0.140 | 0.0300 | 1.00 | 11.4 |
| financialphrasebank | 0.217 | 0.300 | 1.00 | 0.0 |
| imdb | 0.632 | 0.510 | 1.00 | 25.1 |
| manifesto | 0.0171 | 0.00957 | 1.00 | 0.8 |
| massive | 0.264 | 0.0199 | 1.00 | 24.9 |
| rottentomatoes | 0.652 | 0.509 | 1.00 | 29.2 |
| trueteacher | 0.293 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.537 | 0.511 | 1.00 | 5.3 |
| wikitoxic_obscene | 0.619 | 0.514 | 1.00 | 21.6 |
| wikitoxic_threat | 0.288 | 0.360 | 1.00 | 0.0 |
| wikitoxic_toxicaggregated | 0.548 | 0.508 | 1.00 | 8.1 |
| yahootopics | 0.102 | 0.0647 | 1.00 | 4.0 |
| yelpreviews | 0.848 | 0.513 | 1.00 | 68.7 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.4993
- ece
- 0.1684
- accuracy
- 0.5293
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe shipped weights were trained during the session. src/train_cross.py starts from the general MLM encoder answerdotai/
The shipped weights were trained during the session. src/train_cross.py starts from the general MLM encoder answerdotai/ModernBERT-base, pinned to revision 8949b909…, which was confirmed at trajectory#25. It loads that encoder through AutoModelForSequenceClassification with num_labels=2, so the classifier head is newly initialized. The run at trajectory#64 trained for 800 steps, and loss went from 0.687 to 0.039; training.log and train_meta.json match this run. The weight inventory for model/cross/model.safetensors is the ModernBERT layer set plus a newly trained 2x768 classifier and a head. At trajectory#112 the trained cross/ directory was copied into model/cross/ and hashed: sha256 36bb9e6b…, matching training_manifest.json. The later 1600-step v2 run (trajectory#131, #133, #134) was discarded and not shipped.
✓no evaluation data useThe training corpus is synthetic. Label inventories are hand-written in src/make_teacher_jobs.py, and the texts were wri
The training corpus is synthetic. Label inventories are hand-written in src/make_teacher_jobs.py, and the texts were written by the declared teacher, google/gemini-2.5-flash-lite (src/run_teacher.py, training_manifest.json, data_download.log). No load_dataset call or benchmark dataset download appears anywhere in the evidence. /task/public/validation.jsonl was used only to measure accuracy and throughput (trajectory#68, #83, #108), never as training data. The holdout comes from the same synthetic generator.
✓no public split tuningValidation runs measured quality and speed only. The bench_opts runs at trajectory#84–86 used validation texts only as t
Validation runs measured quality and speed only. The bench_opts runs at trajectory#84–86 used validation texts only as timing inputs, and MAX_LENGTH, dtype and compile were chosen for throughput. The temperature fit script (trajectory#111) used the synthetic teacher holdout, not dev data. The checkpoint was chosen by synthetic-holdout behaviour: v2 'overfit the synthetic holdout' (trajectory#134). I found no hyperparameter, threshold, template or checkpoint selection based on public dev scores.
✓no jev reproductionsThe base is answerdotai/ModernBERT-base, a general encoder, and training_manifest.json lists reused_artifacts as []. At
The base is answerdotai/ModernBERT-base, a general encoder, and training_manifest.json lists reused_artifacts as []. At trajectory#31 the participant explicitly rejected public NLI checkpoints and used no JEV, Open-Jev, Jeff or Laya artifacts. The teacher is gemini-2.5-flash-lite, generating texts for locally invented labels. Nothing reproduces the excluded JEV approach.
✓stateless zero shot inferenceserver.py scores each (text, label) pair with the cross-encoder, takes softmax P(entail) and renormalizes across the lab
server.py scores each (text, label) pair with the cross-encoder, takes softmax P(entail) and renormalizes across the labels in each request. It has no label or prediction cache, no cross-request adaptation, no retrieval and no network calls. serve.sh sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE, and /health is static. The dynamic batcher only groups concurrent requests for speed and keeps per-request outputs separate.
✓actual inference uses reviewed weightsserve.sh runs server.py with MODEL_DIR=$ROOT/model. server.py loads the tokenizer from model/tokenizer and AutoModelForS
serve.sh runs server.py with MODEL_DIR=$ROOT/model. server.py loads the tokenizer from model/tokenizer and AutoModelForSequenceClassification from model/cross, whose shipped model.safetensors is the reviewed checkpoint (sha 36bb9e6b…, trajectory#112). At trajectory#119–120 the config was patched to num_labels=2, and a test load confirmed the classifier head matches the raw trained tensors. The official self-check at trajectory#132 served the same folder.
✓harness credentials used only by the harnessTeacher calls went through the Spin OpenRouter proxy. The key was created with `spin openrouter key create` (trajectory#
Teacher calls went through the Spin OpenRouter proxy. The key was created with `spin openrouter key create` (trajectory#31), and TEACHER_BASE_URL was set to api.spin.inductionlabs.com/v1 (trajectory#33). src/run_teacher.py reads only TEACHER_BASE_URL and TEACHER_KEY. There are no ANTHROPIC, OPENAI or XAI key references in the code, and API_KEY has no hits in the trajectory. Hugging Face downloads were unauthenticated or used HF_TOKEN-permitted access.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed36m 11s into the runtook 4m 13s
startup 139.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 139.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0013703471150501864, 0.0009132781347340938, 0.9074197861125317, 0.03709517253586382, 0.008866746814388836, 0.0017588558529480773, 0.003230123894590614, 0.0044788008094945415, 0.002257255238400335, 0.0011542049936070847, 0.01734237538 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 16.4 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.12, p95 0.12
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The classifier is submitted. The official self-check passed before submission.
It is a ModernBERT-base cross-encoder trained for 800 steps on 6,000 synthetic texts written by Gemini 2.5 Flash Lite. Each request scores the text against every supplied label and returns a normalized probability. Nothing is cached across requests.
On the grading-style check, startup took 139 seconds of the 300-second budget, and throughput was 268 documents per second against the 50 required. The held-out labels are longer, and the measured margin is large enough to cover that.