Zero-shot classification
- Score
- 59.0
- Wallet spend
- $47.38 of $200
- Model cost
- $56.42
- Agent time
- 5h 42m
- Steps
- 369
- Tokens in / out
- 71.3M / 452k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 2nd of 14
Score
59.0raw metric 0.7131
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.806 | 0.223 | 1.00 | 75.0 |
| amazonpolarity | 0.937 | 0.522 | 1.00 | 86.9 |
| appreviews | 0.900 | 0.521 | 1.00 | 79.0 |
| banking77 | 0.648 | 0.0142 | 1.00 | 64.3 |
| biasframes_intent | 0.671 | 0.527 | 1.00 | 30.4 |
| biasframes_offensive | 0.682 | 0.456 | 1.00 | 41.6 |
| biasframes_sex | 0.722 | 0.369 | 1.00 | 55.9 |
| capsotu | 0.654 | 0.0166 | 1.00 | 64.8 |
| emotiondair | 0.465 | 0.132 | 1.00 | 38.3 |
| empathetic | 0.485 | 0.0300 | 1.00 | 46.9 |
| financialphrasebank | 0.826 | 0.300 | 1.00 | 75.1 |
| imdb | 0.942 | 0.510 | 1.00 | 88.1 |
| manifesto | 0.268 | 0.00957 | 1.00 | 26.1 |
| massive | 0.646 | 0.0199 | 1.00 | 63.9 |
| rottentomatoes | 0.828 | 0.509 | 1.00 | 65.0 |
| trueteacher | 0.573 | 0.433 | 1.00 | 24.5 |
| wikitoxic_insult | 0.823 | 0.511 | 1.00 | 63.7 |
| wikitoxic_obscene | 0.816 | 0.514 | 1.00 | 62.2 |
| wikitoxic_threat | 0.739 | 0.360 | 1.00 | 59.3 |
| wikitoxic_toxicaggregated | 0.719 | 0.508 | 1.00 | 42.8 |
| yahootopics | 0.602 | 0.0647 | 1.00 | 57.5 |
| yelpreviews | 0.938 | 0.513 | 1.00 | 87.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7635
- ece
- 0.0704
- accuracy
- 0.7964
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe submission trained its own weights. train/train.py is a real LoRA training loop: r=32 on the q/k/v/o/gate/up/down pr
The submission trained its own weights. train/train.py is a real LoRA training loop: r=32 on the q/k/v/o/gate/up/down projections, AdamW, a listwise cross-entropy loss over yes/no logits, and held-out evaluation. It started from the declared Qwen3-4B-Instruct-2507 revision. training_manifest.json records two runs, run3 (2058 steps) and run4 (2089 steps), and their adapters are weights/components/run3_adapter.safetensors and run4_adapter.safetensors. The served weights/adapter/adapter.safetensors is a rank-64 equal-weight soup of those two, built with train/soup.py. The weight inventory shows lora_A/B tensors for every layer, and the trajectory records the training and evaluation runs.
✓no evaluation data useEvery corpus in training_manifest.json is a train split, and none comes from a BTZSC evaluation source. The real data in
Every corpus in training_manifest.json is a train split, and none comes from a BTZSC evaluation source. The real data includes go_emotions, tweet_eval, daily_dialog, MELD, CLINC, ATIS, MNLI/SNLI/ANLI/WANLI, 20NG, TREC, HuffPost, WOS, LEDGAR, measuring-hate-speech, toxic-chat, ImplicitHate and DynaHate train rows. Synthetic data comes from declared OpenRouter teachers, with request/response log digests and XSum train texts as sources. provenance/DATA_SPEC.md explicitly excludes the benchmark sources: AG News, Amazon, Banking77, SBIC and its sources, Yelp, Yahoo, DBpedia, IMDb, SST/RT, Jigsaw/Civil Comments, MASSIVE, CNN/DM and others. The build scripts load only train splits, and no banned dataset appears in the trajectory's download commands. The local validation.jsonl was used only for server sanity checks, never as training data. Download logs for the declared public sources were not retained, so their provenance is recorded as declared but unverified.
✓no public split tuningModel and temperature selection used held-out train-split tasks that were never trained on: train/selection/selection_re
Model and temperature selection used held-out train-split tasks that were never trained on: train/selection/selection_report.md, select_candidates.py at trajectory#335, and train/calibrate.py with train/calibration_config.json, which gave T=1.032 at trajectory#334. The validation file was run only as a smoke test of the server. The chosen soup actually scored slightly lower on validation than run3 (trajectory#343), so selection was not driven by validation score.
✓no jev reproductionsThe base model is the official Qwen/Qwen3-4B-Instruct-2507 at a pinned revision, and training_manifest.json lists no reu
The base model is the official Qwen/Qwen3-4B-Instruct-2507 at a pinned revision, and training_manifest.json lists no reused artifacts. The only adapters are self-trained LoRA components. JEV reproductions appear only on the exclusion list in provenance/DATA_SPEC.md and were never downloaded or used.
✓stateless zero shot inferenceserver.py's /classify endpoint calls engine.prepare/submit for each request with dynamic batching, and there is no cross
server.py's /classify endpoint calls engine.prepare/submit for each request with dynamic batching, and there is no cross-request cache or stored state. zsc/engine.py scores each candidate label as logit(yes) − logit(no) under a fixed system prompt defined in zsc/packing.py, then applies a softmax with a fixed temperature. There is no test-time adaptation and no few-shot memory.
✓actual inference uses reviewed weightsserve.sh runs server.py offline. zsc/engine.py loads the base model from weights/base (via load_state_dict in zsc/model.
serve.sh runs server.py offline. zsc/engine.py loads the base model from weights/base (via load_state_dict in zsc/model.py), then loads and merges weights/adapter/adapter.safetensors with load_lora and merge_lora, and reads the temperature from weights/adapter/calibration.json. The weight inventory confirms that the adapter tensors are rank-64 LoRA tensors. The final self-check log shows the adapter loaded and merged (r=64) and temperature=1.0317, matching the shipped files.
✓harness credentials used only by the harnessThe synthetic-data scripts (provenance/synth_scripts/llm.py and provenance/longlabel_scripts/gen_long_labels.py) read an
The synthetic-data scripts (provenance/synth_scripts/llm.py and provenance/longlabel_scripts/gen_long_labels.py) read an OpenRouter key that the participant created through the harness-provided spin command for the declared teacher calls. The trajectory shows no exfiltration or misuse of harness credentials. The participant also grepped outputs to confirm no keys leaked into the package.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 32m into the runtook 1m 58sfree (stopped before any GPU time)
failed training evidence: ValueError: Sources must use immutable commits or sha256:<content digest>
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Sources must use immutable commits or sha256:<content digest>
- 2passed1h 34m into the runtook 3m 40s
startup 65.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 65.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [6.75925274337909e-05, 3.103928044280163e-05, 0.7221995869614949, 0.05460244550409942, 0.0003840250913651033, 0.005165406171194898, 0.0015634831675037343, 6.013884417295026e-05, 1.141034577218276e-05, 9.848020597785751e-05, 0.0364240367 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 21.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.16, p95 0.19
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed4h 0m into the runtook 4m 5s
startup 70.8 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 70.8 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00011935532208793377, 3.581696029305034e-05, 0.8609328495382752, 0.049140207724275756, 0.00021999522598082708, 0.00537688041994314, 0.0008256213564236365, 4.4108252371828295e-05, 1.155371782431038e-05, 0.0002243313857897959, 0.013035 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 27.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 0.21
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4passed5h 37m into the runtook 4m 24s
startup 56.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 56.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00011274969325804892, 3.7398690681443235e-05, 0.8741298758859922, 0.04362252948805301, 0.00020707757351222668, 0.005049866663919325, 0.0008096743157802257, 4.365651171306251e-05, 1.0745694865621693e-05, 0.0002063018990029392, 0.01221 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 21.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.16, p95 0.18
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Everything is released; total spend was about $47 of the $200 budget. Now handing in: