Zero-shot classification
- Score
- 57.4
- Wallet spend
- $23.01 of $50
- Model cost
- $8.76
- Agent time
- 4h 57m
- Steps
- 368
- Tokens in / out
- 147.6M / 476k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
57.4raw metric 0.6901
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.868 | 0.223 | 1.00 | 83.0 |
| amazonpolarity | 0.918 | 0.522 | 1.00 | 82.8 |
| appreviews | 0.885 | 0.521 | 1.00 | 75.9 |
| banking77 | 0.598 | 0.0142 | 1.00 | 59.2 |
| biasframes_intent | 0.638 | 0.527 | 1.00 | 23.4 |
| biasframes_offensive | 0.670 | 0.456 | 1.00 | 39.4 |
| biasframes_sex | 0.766 | 0.369 | 1.00 | 62.8 |
| capsotu | 0.563 | 0.0166 | 1.00 | 55.5 |
| emotiondair | 0.442 | 0.132 | 1.00 | 35.7 |
| empathetic | 0.401 | 0.0300 | 1.00 | 38.2 |
| financialphrasebank | 0.790 | 0.300 | 1.00 | 70.0 |
| imdb | 0.915 | 0.510 | 1.00 | 82.7 |
| manifesto | 0.293 | 0.00957 | 1.00 | 28.6 |
| massive | 0.621 | 0.0199 | 1.00 | 61.3 |
| rottentomatoes | 0.847 | 0.509 | 1.00 | 68.8 |
| trueteacher | 0.201 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.811 | 0.511 | 1.00 | 61.3 |
| wikitoxic_obscene | 0.796 | 0.514 | 1.00 | 58.0 |
| wikitoxic_threat | 0.824 | 0.360 | 1.00 | 72.6 |
| wikitoxic_toxicaggregated | 0.793 | 0.508 | 1.00 | 57.9 |
| yahootopics | 0.591 | 0.0647 | 1.00 | 56.3 |
| yelpreviews | 0.953 | 0.513 | 1.00 | 90.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7556
- ece
- 0.1001
- accuracy
- 0.7911
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe trained weights were produced during the task. train_zsc.py starts from Qwen/Qwen3-4B-Base at pinned revision 906bfd
The trained weights were produced during the task. train_zsc.py starts from Qwen/Qwen3-4B-Base at pinned revision 906bfd4b and trains a PEFT LoRA adapter (r=32 on q/k/v/o/gate/up/down), plus a marker vector initialised from the newline embedding and a linear head initialised from normal(0.02). Its save() layout (lora_A/B, marker, head.weight/bias) matches the adapter.safetensors tensor inventory. training_manifest.json lists reused_artifacts as [] and points to train_run2.log (5777 steps). At trajectory#314 and #362 the participant checked that the adapter's sha256 (34974d1f…) matches the manifest.
✓no evaluation data usefetch_public.py downloads only train splits, each pinned to a revision: DBpedia-14, 20 Newsgroups, BBC News, CLINC, ATIS
fetch_public.py downloads only train splits, each pinned to a revision: DBpedia-14, 20 Newsgroups, BBC News, CLINC, ATIS, GoEmotions, TweetEval configs, Twitter financial news, News Category, MultiNLI, VitaminC and ToxiGen. None of these is a BTZSC-excluded source. build_pool_v2.py reads only those raw files plus teacher-generated synthetic tasks. A search found no excluded corpora (AG News, IMDb, Yelp, Banking77, etc.) anywhere except in exclusion statements. At trajectory#16 the participant explicitly dropped DAIR Emotion as excluded. The teacher is qwen3-235b, declared and reached through Spin.
✓no public split tuningcalibration.json (T=1.1, validation_split_used false) and data_manifest.json say the temperature and run selection were
calibration.json (T=1.1, validation_split_used false) and data_manifest.json say the temperature and run selection were chosen on train-split dev holdouts plus a TweetEval stance train-split OOD check. The trajectory shows T=1.1 was fixed at trajectory#305, before validation.jsonl was uploaded at #306 and used only in the final validate.py sanity runs (#310–#313, macro-F1 0.988). After that, only documentation was changed (#314), and the serving files' hashes were confirmed unchanged at #361.
✓no jev reproductionsSearches for Jeff, Laya and JEV found them only in the policy text and in the participant's own exclusion notes. The arc
Searches for Jeff, Laya and JEV found them only in the policy text and in the participant's own exclusion notes. The architecture is an original Qwen3-4B-Base + LoRA + marker-token scoring head written in zsc_runtime.py and train_zsc.py, and the manifest lists no reused artifacts.
✓stateless zero shot inferenceserve.py handles each /classify request on its own. It encodes the text and that request's labels (zsc_runtime.Encoder),
serve.py handles each /classify request on its own. It encodes the text and that request's labels (zsc_runtime.Encoder), scores the labels in a forward pass that may be micro-batched with other independent requests, and applies a fixed temperature softmax. There is no caching, retrieval, cross-request state, online learning or external service call. Environment variables set HF offline mode.
✓actual inference uses reviewed weightsserve.sh runs serve.py, which builds a Scorer from the shipped base/ shards and adapter.safetensors. zsc_runtime.Scorer
serve.sh runs serve.py, which builds a Scorer from the shipped base/ shards and adapter.safetensors. zsc_runtime.Scorer merges every lora_A/lora_B pair into the base weights and raises an error if the merged-module count is wrong. It also uses state['marker'] and head.weight/head.bias from the same adapter file, whose tensor inventory matches the trained layout. At trajectory#361 the participant confirmed the serving files match the tested sources.
✓harness credentials used only by the harnessThe teacher key came from `spin openrouter key create` (trajectory#16, #17). It was stored in spin_llm.json and used by
The teacher key came from `spin openrouter key create` (trajectory#16, #17). It was stored in spin_llm.json and used by llm_client.py only against the Spin OpenAI-compatible endpoint, which the policy allows. No harness or Anthropic credentials appear in the code or the trajectory, and the serving code makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 43m into the runtook 4m 18s
startup 62.8 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 62.8 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [3.167364680914993e-06, 4.229897904176887e-07, 0.9997166829592256, 2.1820770279804785e-05, 3.5484621463840876e-06, 2.8875684472112622e-05, 4.077574643098526e-05, 9.99346268232425e-07, 1.2905048148879514e-07, 9.060709871685891e-07, 0.000 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 36.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.26, p95 0.38
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed3h 55m into the runtook 4m 58s
startup 61.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 61.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [3.1618284507207596e-05, 6.089002494836764e-06, 0.9983124087254189, 0.00015336116541455653, 3.4698373527593844e-05, 0.00019286668177156578, 0.00025578661629912527, 1.2303930756433391e-05, 2.305238916824917e-06, 1.1356046360300985e-05, 0 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 35.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.26, p95 0.37
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The classifier is submitted (11:07 UTC). Both official selfchecks passed on the submitted tree, which is the same serving code and calibration.
What's submitted
- Model: Qwen3-4B-Base with a trained LoRA adapter (r=32), a learned marker vector, and a linear scoring head. The text and all label descriptions go into one prompt. Each label is scored at its own marker position, and a softmax runs over the request's labels, so any label set of 2–256 works.
- Calibration: temperature 1.1. I chose it on held-out data only, weighting in-distribution dev and an out-of-distribution stance check equally, since hidden datasets are probably harder than my dev split.
- Serving:
serve.shinstalls the vendored cp312 wheels into/tmpand starts two worker processes sharing port 8000.
Gates from frontier selfcheck
- Startup: 61–63 s of 300 s.
- Throughput: 122 docs/s on resampled public documents, against a 50 docs/s requirement. The report warns that held-out requests are about 1.75× larger in bytes, so I expect roughly 70 docs/s there. That is still above the gate, but with less margin than the public number suggests.
Local numbers (not validation-selected)
- In-distribution dev (train-split holdouts and held-out synthetic tasks): about 0.85 with sampled label descriptions and 0.84 with bare category names.
- Out-of-distribution TweetEval stance: about 0.53. This is a rough proxy for harder zero-shot tasks.
- Public
validate.pyrun, once, as a sanity check after choosing T: macro-F1 0.988, ECE 0.009. This split is DBpedia test data, and DBpedia train is in my training pool, so it is optimistic.
Compliance and provenance
- No split of any excluded source was used, and the validation split was used only for that final check.
- Training data came from public train splits (pinned SHAs in the manifests) plus synthetic tasks from a teacher model reached through spin. The synthetic corpus is digested, not shipped.
training/holds the code, configs, logs, and the reconstructed teacher requests.
Things to know
- I tested a second backbone, the post-trained Qwen3-4B, under the same recipe. It was worse in distribution, so I kept the base model.
- Run 1 had three duplicated synthetic IDs, which may have caused about 36 wrong gold labels. The submitted run rebuilt its pool keyed by ID and is not affected. The docs note this.
- Test-time augmentation over two label orders gained only about 0.5 points at twice the inference cost, so I didn't use it.
- Budget: $23 of $50 spent.