Zero-shot classification
- Score
- 57.7
- Wallet spend
- $14.11 of $50
- Model cost
- $33.36
- Agent time
- 3h 32m
- Steps
- 208
- Tokens in / out
- 44.0M / 303k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 2nd of 14
Score
57.7raw metric 0.688
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.856 | 0.223 | 1.00 | 81.5 |
| amazonpolarity | 0.940 | 0.522 | 1.00 | 87.5 |
| appreviews | 0.878 | 0.521 | 1.00 | 74.4 |
| banking77 | 0.582 | 0.0142 | 1.00 | 57.6 |
| biasframes_intent | 0.764 | 0.527 | 1.00 | 50.2 |
| biasframes_offensive | 0.740 | 0.456 | 1.00 | 52.2 |
| biasframes_sex | 0.536 | 0.369 | 1.00 | 26.4 |
| capsotu | 0.610 | 0.0166 | 1.00 | 60.3 |
| emotiondair | 0.429 | 0.132 | 1.00 | 34.2 |
| empathetic | 0.402 | 0.0300 | 1.00 | 38.4 |
| financialphrasebank | 0.687 | 0.300 | 1.00 | 55.3 |
| imdb | 0.942 | 0.510 | 1.00 | 88.2 |
| manifesto | 0.275 | 0.00957 | 1.00 | 26.8 |
| massive | 0.635 | 0.0199 | 1.00 | 62.8 |
| rottentomatoes | 0.870 | 0.509 | 1.00 | 73.6 |
| trueteacher | 0.212 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.839 | 0.511 | 1.00 | 67.1 |
| wikitoxic_obscene | 0.805 | 0.514 | 1.00 | 59.9 |
| wikitoxic_threat | 0.831 | 0.360 | 1.00 | 73.6 |
| wikitoxic_toxicaggregated | 0.794 | 0.508 | 1.00 | 58.1 |
| yahootopics | 0.563 | 0.0647 | 1.00 | 53.2 |
| yelpreviews | 0.944 | 0.513 | 1.00 | 88.5 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7458
- ece
- 0.0911
- accuracy
- 0.7775
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingFour rank-32 LoRA runs (pilot, A, A2 and B) plus a trainable head were trained with training/train.py during this sessio
Four rank-32 LoRA runs (pilot, A, A2 and B) plus a trainable head were trained with training/train.py during this session. The trajectory shows the train.py launches. The weight inventories for weights/trained/adapter_*.safetensors contain real LoRA A/B tensors and a head. combine_adapters.py merges them into the rank-128 weights/adapter.safetensors at a weight of 0.1625 each, which matches training_manifest.json. The submission learns parameters, not just a temperature.
✓no evaluation data usetraining_manifest.json and collect_datasets.py declare pinned train-split HF sources: DBpedia, 20NG, HuffPost, TREC, Lex
training_manifest.json and collect_datasets.py declare pinned train-split HF sources: DBpedia, 20NG, HuffPost, TREC, LexGLUE, CLINC, ATIS, tweet_eval, GoEmotions, NLI sets and others. None of them is an excluded BTZSC source. collect_datasets.py enforces a FORBIDDEN_PATTERNS guard and downloads train files only. The synthetic data in synth_generate*.py is newly generated from label names by the declared OpenRouter teacher; the consistency data uses invented articles, not CNN/DM. Every corpus's provenance is declared; no download log was retained, so the declared HF sources are unverified, but that does not fail this check.
✓no public split tuningThe temperature and the soup choice were selected with calibrate.py on train splits of held-out datasets (bbc-news, snip
The temperature and the soup choice were selected with calibrate.py on train splits of held-out datasets (bbc-news, snips and others), not on validation.jsonl. In the trajectory, validation.jsonl was used only to run validate.py as a server check (steps 63-95) and once to compute average request size for a throughput stress test (step 195). Neither use feeds training, calibration or selection. No source file reads it apart from the guard comment.
✓no jev reproductionsreused_artifacts in training_manifest.json is empty. The base is Qwen3-4B-Instruct-2507 at a pinned revision, loaded fro
reused_artifacts in training_manifest.json is empty. The base is Qwen3-4B-Instruct-2507 at a pinned revision, loaded from weights/base. collect_datasets.py guards against jev, laya and typesafe sources. No Jeff, Laya, Open-Jev, jev-4b or typesafe-diffusion-lab artifacts appear in the code, the weights or the trajectory.
✓stateless zero shot inferenceserver.py scores each request independently. It encodes, packs and runs the forward pass, then applies a per-document so
server.py scores each request independently. It encodes, packs and runs the forward pass, then applies a per-document softmax at T=1.4. There is no cross-request cache, no benchmark or dataset detection and no dataset-specific logic. Batching only groups queued independent requests, using block masks that isolate each document. zsc_model.py has no caches either.
✓actual inference uses reviewed weightsserve.sh runs server.py with the weights directory. server.py reads weights/serving_config.json (adapter.safetensors, sc
serve.sh runs server.py with the weights directory. server.py reads weights/serving_config.json (adapter.safetensors, scale 1.0, temperature 1.4). It then calls zsc_model.load_scorer on the weights/base shards, load_adapter and merge_lora, and FastScorer builds from those merged weights. weights/adapter.safetensors is the 4-run soup copied from the export (trajectory#181). There are no external services and no changes to grading files.
✓harness credentials used only by the harnessThe only API credential used is the Spin OpenRouter key created with `spin openrouter key create` and stored in /workspa
The only API credential used is the Spin OpenRouter key created with `spin openrouter key create` and stored in /workspace/work/orkey.json (trajectory#64 area, steps 63-78). synth_generate.py and synth_generate2.py use that key. Searches found no use of ANTHROPIC or harness API keys or base URLs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 53m into the runtook 6m 46s
startup 138.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 138.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0005560021657458141, 0.0003564292790930054, 0.983849626799893, 0.0032781944865555484, 0.0010752522096066101, 0.00047306817672868517, 0.0003184279438450634, 0.00036842149962678256, 0.00040158069031349513, 0.0003127779026103633, 0.0011 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 38.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.27, p95 1.13
- ✓batch responses · all 4400 responses parsed; 11 succeeded on a retry
- ✓healthy after requests
- 2passed3h 26m into the runtook 6m 20s
startup 124.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 124.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0005357671009447409, 0.00034303458250459066, 0.9838433491779911, 0.0034401913931426083, 0.0010086349191678625, 0.00045849476322975397, 0.0003001139482190298, 0.0003599662262154746, 0.00039299208056997685, 0.0003018761302338254, 0.001 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 33.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.24, p95 0.25
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The folder is unchanged since the passing selfcheck #2 (only read-only checks ran afterwards). Submitting.