Zero-shot classification
- Score
- 63.6
- Wallet spend
- $34.48 of $50
- Model cost
- $20.30
- Agent time
- 5h 55m
- Steps
- 430
- Tokens in / out
- 42.7M / 370k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 1st of 14
Score
63.6raw metric 0.7399
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.862 | 0.223 | 1.00 | 82.3 |
| amazonpolarity | 0.931 | 0.522 | 1.00 | 85.6 |
| appreviews | 0.909 | 0.521 | 1.00 | 81.1 |
| banking77 | 0.644 | 0.0142 | 1.00 | 63.9 |
| biasframes_intent | 0.792 | 0.527 | 1.00 | 56.0 |
| biasframes_offensive | 0.755 | 0.456 | 1.00 | 55.1 |
| biasframes_sex | 0.568 | 0.369 | 1.00 | 31.4 |
| capsotu | 0.670 | 0.0166 | 1.00 | 66.5 |
| emotiondair | 0.444 | 0.132 | 1.00 | 35.9 |
| empathetic | 0.518 | 0.0300 | 1.00 | 50.3 |
| financialphrasebank | 0.857 | 0.300 | 1.00 | 79.6 |
| imdb | 0.931 | 0.510 | 1.00 | 85.9 |
| manifesto | 0.340 | 0.00957 | 1.00 | 33.4 |
| massive | 0.702 | 0.0199 | 1.00 | 69.5 |
| rottentomatoes | 0.870 | 0.509 | 1.00 | 73.5 |
| trueteacher | 0.661 | 0.433 | 1.00 | 40.2 |
| wikitoxic_insult | 0.849 | 0.511 | 1.00 | 69.0 |
| wikitoxic_obscene | 0.850 | 0.514 | 1.00 | 69.2 |
| wikitoxic_threat | 0.771 | 0.360 | 1.00 | 64.3 |
| wikitoxic_toxicaggregated | 0.822 | 0.508 | 1.00 | 63.9 |
| yahootopics | 0.592 | 0.0647 | 1.00 | 56.3 |
| yelpreviews | 0.939 | 0.513 | 1.00 | 87.5 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7844
- ece
- 0.0611
- accuracy
- 0.8152
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe public Qwen/Qwen3-4B-Base checkpoint was fetched from HF with snapshot_download during the session (trajectory#129,
The public Qwen/Qwen3-4B-Base checkpoint was fetched from HF with snapshot_download during the session (trajectory#129, trajectory#130). training/train.py then ran PEFT LoRA r=64 training with a score_w head, merge_and_unload and save. The run reached step 1434 (training/logs/training.log). The trajectory shows the trained output (sha 0c0a16db…) being installed as model/model.safetensors (trajectory#415, trajectory#416). The inventory shows the Qwen3 BF16 tensors plus the trained score_w F32 vector, and the sha matches training_manifest.json.
✓no evaluation data useThe download.py FILES list (trajectory#20, unchanged in later steps) and provenance_real.json contain only train splits
The download.py FILES list (trajectory#20, unchanged in later steps) and provenance_real.json contain only train splits of allowed datasets: 20 Newsgroups, TREC, CLINC, GoEmotions, TweetEval, HateXplain, ETHOS, MNLI, SNLI, ANLI, WANLI, VitaminC, BBC News, tweet topic, financial tweets and XSum train, among others. XSum is explicitly allowed. Excluded datasets such as ag_news, dair-ai/emotion, sst2 and dbpedia_14 were only probed for metadata in steps 18–19 and never downloaded. Synthetic data came from generic family and domain seeds (seeds.py, gen_synth.py), and fact data from XSum train (gen_fact.py). Every source is declared; the download commands and logs are retained.
✓no public split tuning/task/public/validation.jsonl was used only to run /task/validate.py against a local server as a server check (trajector
/task/public/validation.jsonl was used only to run /task/validate.py against a local server as a server check (trajectory#146) and in a short head to see the format. Calibration fitting (T=1.15) and the 4B-vs-1.7B model selection used only stage/calib.jsonl. build_calib.py builds that file from held-out train-split pools, ANLI train, an XSum holdout and a synthetic holdout (fit_calib2.py, calibration_fit.log, trajectory#410, trajectory#417).
✓no jev reproductionstraining_manifest.json lists reused_artifacts as []. The base model is the official Qwen3-4B-Base at a pinned revision (
training_manifest.json lists reused_artifacts as []. The base model is the official Qwen3-4B-Base at a pinned revision (architecture.json). Searching the trajectory for Jeff, Laya and JEV found no use. All training was done from scratch in-session with training/train.py.
✓stateless zero shot inferencesrc/server.py handles each /classify request by encoding the prompt, sending it through a dynamic batcher, computing raw
src/server.py handles each /classify request by encoding the prompt, sending it through a dynamic batcher, computing raw scores and returning a softmax over labels. It makes no external network calls and has no cross-request cache or memory. Warm-up uses synthetic text. src/engine.py and src/lw.py only perform the forward pass.
✓actual inference uses reviewed weightsThe serving path is serve.sh → `python src/server.py --model $HERE/model` → src/engine.py, which loads tokenizer.json, l
The serving path is serve.sh → `python src/server.py --model $HERE/model` → src/engine.py, which loads tokenizer.json, lw_format.json, calibration.json and model/model.safetensors through lw.load_model. load_model reads every tensor, including score_w, and the scoring computes rms_unit(h)·score_w. The shipped sha 0c0a16db… matches the trained output installed in trajectory#416.
✓harness credentials used only by the harnessTeacher LLM calls in training/data_pipeline/gen/llm.py read /workspace/work/.spin_key.json, a key created with `spin ope
Teacher LLM calls in training/data_pipeline/gen/llm.py read /workspace/work/.spin_key.json, a key created with `spin openrouter key create`, and post to the Spin OpenRouter proxy (api.spin.inductionlabs.com). The policy allows this. Trajectory searches found no ANTHROPIC, OPENAI, OPENROUTER or GEMINI harness key use, no echoing of *_BASE_URL (it appears only in spin help text), and no extra claude, codex or gemini CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 16m into the runtook 4m 34s
startup 90.6 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 90.6 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0013925166067013902, 0.0017296078588210112, 0.7037204723134022, 0.009417872914331646, 0.026857451699181925, 0.008577268942758847, 0.007393840596536408, 0.009746805846612966, 0.020276108635495872, 0.010808835878577956, 0.0314730305885 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 30.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.22, p95 0.23
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed3h 10m into the runtook 3m 58s
startup 103.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 103.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00019383280967513774, 0.00018748856993840008, 0.9144594933022376, 0.00458211643143132, 0.000585697450068709, 0.0005623598726569113, 0.00043735265759943596, 0.00023267029592447512, 6.704309725779711e-05, 0.0001235394984135428, 0.02334 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 18.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.13, p95 0.14
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed5h 49m into the runtook 4m 58s
startup 97.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 97.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0004110734682798505, 0.00024316036604098905, 0.9171181866417215, 0.003408725459218084, 0.0005919009188137879, 0.0011694335210621289, 0.0008075043895162549, 0.00016969358793182088, 7.880990767521305e-05, 0.00014458432947812952, 0.0401 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 31.4 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.23, p95 0.24
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
I submitted the zero-shot classifier. No held-out score is available yet, and the self-check reports none. It is a Qwen3-4B-Base model trained as a listwise classifier, served by /workspace/output/serve.sh.
- Self-check: the last attempt on the final submission passed. Startup was 97.7 s of the 300 s allowed and the throughput gate passed.
- Throughput margin: I also load-tested the final server on the rented GPU (power-capped, about half as fast as the grading H100). With a synthetic BTZSC-like request mix it did 46.7 docs/s with no failures. On the grading H100 that should be roughly 90 docs/s against the 50 required, but this is an estimate.
- Calibration: the temperature is T=1.15. On training-split calibration rows (held out from training), the 4B scored 0.7188 mean macro-F1 × (1−ECE) at T=1.0. The score is flat for T between 1.0 and 1.3. I picked 1.15 because the noisy-label sources (goemo, te_off) were over-confident at 1.0, and held-out data is likely harder than the calibration rows.
- Compliance: I did not use the public validation rows, any excluded dataset, or JEV artifacts. The trained weights hash is recorded in
training_manifest.json, andarchitecture.jsonand the training code, logs and config are in the output. - Smaller model: I also trained a Qwen3-1.7B-Base model for comparison, but it was not shipped. The 4B scored higher on the calibration rows (0.7188 vs 0.7081 at its best temperature).
- Training log gap: the 4B training log shows an elapsed-time jump because the run was paused for about 40 minutes while the 1.7B run finished on the same GPU. The README explains this.
The session is over.