Zero-shot classification
- Score
- 59.3
- Wallet spend
- $8.95 of $10
- Model cost
- $16.20
- Agent time
- 3h 13m
- Steps
- 191
- Tokens in / out
- 43.6M / 227k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 3rd of 14
Score
59.3raw metric 0.7174
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.894 | 0.223 | 1.00 | 86.4 |
| amazonpolarity | 0.939 | 0.522 | 1.00 | 87.3 |
| appreviews | 0.891 | 0.521 | 1.00 | 77.2 |
| banking77 | 0.649 | 0.0142 | 1.00 | 64.4 |
| biasframes_intent | 0.646 | 0.527 | 1.00 | 25.2 |
| biasframes_offensive | 0.664 | 0.456 | 1.00 | 38.2 |
| biasframes_sex | 0.588 | 0.369 | 1.00 | 34.7 |
| capsotu | 0.661 | 0.0166 | 1.00 | 65.5 |
| emotiondair | 0.438 | 0.132 | 1.00 | 35.2 |
| empathetic | 0.546 | 0.0300 | 1.00 | 53.2 |
| financialphrasebank | 0.787 | 0.300 | 1.00 | 69.6 |
| imdb | 0.916 | 0.510 | 1.00 | 82.8 |
| manifesto | 0.336 | 0.00957 | 1.00 | 32.9 |
| massive | 0.680 | 0.0199 | 1.00 | 67.3 |
| rottentomatoes | 0.844 | 0.509 | 1.00 | 68.3 |
| trueteacher | 0.575 | 0.433 | 1.00 | 25.1 |
| wikitoxic_insult | 0.769 | 0.511 | 1.00 | 52.6 |
| wikitoxic_obscene | 0.849 | 0.514 | 1.00 | 69.0 |
| wikitoxic_threat | 0.734 | 0.360 | 1.00 | 58.4 |
| wikitoxic_toxicaggregated | 0.832 | 0.508 | 1.00 | 65.9 |
| yahootopics | 0.607 | 0.0647 | 1.00 | 58.0 |
| yelpreviews | 0.937 | 0.513 | 1.00 | 87.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7723
- ece
- 0.0749
- accuracy
- 0.8027
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe LoRA adapter (r=32) on Qwen3-4B-Instruct-2507 and the PointerHead were trained during the session. training/train.py
The LoRA adapter (r=32) on Qwen3-4B-Instruct-2507 and the PointerHead were trained during the session. training/train.py was run as run1 for 2577 steps, with logs in training/run1. Trajectory#173 copies the run1 adapter and head into model/. Their SHAs (adapter bf1cd952…, head 7e94b5b0…) match the hashes recorded in training_manifest.json. The manifest's reused_artifacts list is empty.
✓no evaluation data usebuild_real.py loads only the train split of each public dataset, at pinned revisions; the hf_hub_download files are also
build_real.py loads only the train split of each public dataset, at pinned revisions; the hf_hub_download files are also train files (atis_train.csv, mtop en/train.jsonl, vitaminc train.jsonl, anli train_r1-3). The only GLUE config is qqp train. Excluded sources are not used: SST, MR, subj, CNN/DailyMail derivatives and BTZSC sources. data_manifest.json lists the BTZSC sources, DBpedia and validation.jsonl under never_used. The consistency pairs use XSum train, which is allowed, and the synthetic tasks are invented by deepseek-v4-flash rather than copied from a dataset.
✓no public split tuningCheckpoint and temperature (T=1.1) were chosen on a held-out dev set built from 16 training-split sources (DEV_SOURCES i
Checkpoint and temperature (T=1.1) were chosen on a held-out dev set built from 16 training-split sources (DEV_SOURCES in build_mix.py), per eval_dev.py, select_temperature.py and selection.json (trajectory#159-#171). The run1-vs-run2 choice was also made on dev (trajectory#173). The provided validator was run on public/validation.jsonl, as the task prompt instructs, but only as a smoke, throughput and final check. Nothing was selected or calibrated on its results.
✓no jev reproductionsThe backbone is the official Qwen3-4B-Instruct-2507 at a pinned revision. The adapter and head were trained from scratch
The backbone is the official Qwen3-4B-Instruct-2507 at a pinned revision. The adapter and head were trained from scratch in this session. training_manifest.json declares no reused artifacts, and the teacher model was used only to generate data. No reuse of a prior JEV submission or checkpoint was found.
✓stateless zero shot inferenceserver/app.py scores each request on its own: it formats the text and labels (zsc_format.py), runs a packed forward pass
server/app.py scores each request on its own: it formats the text and labels (zsc_format.py), runs a packed forward pass and returns softmax(score/T). There is no cache across requests, no external calls and no labeled examples from the evaluation task. The fallbacks are deterministic: uniform probabilities on error, and [1.0] for a single label.
✓actual inference uses reviewed weightsserve.sh runs server/app.py. The Engine in app.py loads the pinned backbone and merges model/adapter/adapter_model.safet
serve.sh runs server/app.py. The Engine in app.py loads the pinned backbone and merges model/adapter/adapter_model.safetensors through lora_merge.py. It then loads the PointerHead from model/head.safetensors, reads T from model/classifier.json and serves through PackedQwen3. These are the same files whose hashes appear in the manifest and the trajectory.
✓harness credentials used only by the harnessThe only external key the participant used is an OpenRouter key issued by the Spin CLI (`spin openrouter key create`, tr
The only external key the participant used is an OpenRouter key issued by the Spin CLI (`spin openrouter key create`, trajectory#18-31), read by or_client.py from or_key.json. That is permitted. Searches found no participant use of ANTHROPIC_API_KEY, `claude -p` or the judge key. The only MLE_JUDGE_API_KEY match is validator code shown in tool output (trajectory steps 76-98).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 55m into the runtook 1m 33sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2failed1h 57m into the runtook 54sfree (stopped before any GPU time)
failed training evidence: ValueError: Evidence path missing, symlinked or outside submission
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Evidence path missing, symlinked or outside submission
- 3passed1h 58m into the runtook 3m 20s
startup 29.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 29.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00038159493128940027, 0.0001496982988492804, 0.8068599715812456, 0.013785124624220402, 0.004185235576433226, 0.0013362335295645162, 0.0013603982629243967, 0.00047513886898580796, 0.0002406103286483222, 0.0001487347531332498, 0.022192 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 31.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.23, p95 0.24
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4passed3h 9m into the runtook 3m 9s
startup 32.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 32.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [2.234706369646679e-05, 5.574488788505807e-06, 0.8807965251532065, 0.0018752843904471445, 5.4728568665436126e-05, 0.0007180531325002634, 0.0006221235416745465, 2.283781925826039e-05, 1.927218794016612e-06, 1.3623320225751364e-05, 0.0186 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 31.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.23, p95 0.24
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Updating memory with the final outcome, then submitting: