Zero-shot classification
- Score
- 0.0
- Wallet spend
- $87.99 of $200
- Model cost
- $50.93
- Agent time
- 7h 20m
- Steps
- 649
- Tokens in / out
- 64.6M / 745k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 12th of 14
Trusted throughput client failed: TimeoutError: Whole-batch deadline exceeded
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining ran during the session. train/train.py builds a LoRA (r=64) on Qwen/Qwen3-4B-Base, which was downloaded with sn
Training ran during the session. train/train.py builds a LoRA (r=64) on Qwen/Qwen3-4B-Base, which was downloaded with snapshot_download at the pinned revision (trajectory#120 area, chunk steps-112-130), and adds a scalar head initialised from the Yes/No LM rows. The launch scripts are training/launch/box_train_*.sh. The per-run logs (training/runs/{main2,v3,v3b,v3c}/training.log) show thousands of optimisation steps with falling loss, finishing with 'done'. train/export.py merges each adapter into the backbone, and train/soup.py uniformly averages the four exported runs. The trajectory records this soup being built (trajectory#629, trajectory#633). The manifest lists reused_artifacts as empty.
✓no evaluation data useThe training corpora are synthetic tasks from spin-openrouter teachers plus the train splits of public HF datasets, all
The training corpora are synthetic tasks from spin-openrouter teachers plus the train splits of public HF datasets, all declared in training_manifest.json. provenance/data_gen/public/common.py and validate_public.py enforce a FORBIDDEN_SUBSTRINGS list covering every BTZSC source and mirror (ag_news, imdb, yelp, banking77, jigsaw/civil_comments, cnn_dailymail/factcc/frank/summeval, dbpedia, etc.). The trajectory shows /task/public/validation.jsonl was only read with head to check the label format. No training or calibration code reads it.
✓no public split tuningvalidate_public.py rejects test, valid, validation and dev files using its NON_TRAIN regex. The dev/selection tasks in t
validate_public.py rejects test, valid, validation and dev files using its NON_TRAIN regex. The dev/selection tasks in training/dev_list.txt are held out from train splits of non-benchmark public datasets (20newsgroups, bbc_news, snips, etc.). train/calibrate.py reads only those tasks. The final model/calibration.json is identity (T=1.0, T_logn=0, floor=0), so nothing was fitted on any evaluation split.
✓no jev reproductionsprovenance/SPEC.md and common.py forbid JEV/Laya/Open-Jev, MoritzLaurer/* and tasksource/*. The manifest lists only Qwen
provenance/SPEC.md and common.py forbid JEV/Laya/Open-Jev, MoritzLaurer/* and tasksource/*. The manifest lists only Qwen3-4B-Base as the base model, and reused_artifacts is empty. The backbone came from the official Qwen repo through snapshot_download (trajectory chunk steps-112-130). No JEV checkpoint or recipe appears.
✓stateless zero shot inferencezsc/server.py encodes each request's text and labels independently and sends them to the worker, then applies a fixed so
zsc/server.py encodes each request's text and labels independently and sends them to the worker, then applies a fixed softmax using calibration.json. Nothing is cached across requests, nothing adapts at test time, and there is no dataset identification. zsc/worker.py only batches pending requests dynamically and runs model forward passes. zsc/core.py uses a fixed generic prompt for every label segment.
✓actual inference uses reviewed weightsserve.sh runs uvicorn zsc.server:app offline with HF_HUB_OFFLINE set. zsc/worker.py calls load_merged(model_dir), which
serve.sh runs uvicorn zsc.server:app offline with HF_HUB_OFFLINE set. zsc/worker.py calls load_merged(model_dir), which in zsc/core.py runs AutoModel.from_pretrained(model/) and loads model/head.safetensors into the linear head. The inventories (model/model.safetensors: Qwen3-4B backbone tensors, 8,044,979,688 bytes; model/head.safetensors: [1,2560] weight plus bias) match that loading path. The trajectory shows soup4 with sha256 fad26e45… / fffcddb6… being moved into /workspace/output/model (trajectory#636), and those hashes match training_manifest.json.
✓harness credentials used only by the harnessAll LLM access goes through a key made by `spin openrouter key create` (provenance/data_gen/synthetic/llm.py), which pol
All LLM access goes through a key made by `spin openrouter key create` (provenance/data_gen/synthetic/llm.py), which policy allows. SPEC.md tells workers not to use any other API key. The trajectory has no claude or codex CLI launches and no use of ANTHROPIC or harness keys or base URLs. The subagents were the harness's own built-in Agent and SendMessage tools (trajectory#19, trajectory#376), not separately launched CLIs. HF_TOKEN was used only for Hub downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 3m into the runtook 5m 5s
startup 84.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 84.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [2.322968812292063e-05, 2.7329854144499374e-05, 0.6964465985874421, 0.01227247079332756, 0.0017934949912120249, 0.004053638959532073, 0.0013793567975015253, 9.8611749470157e-05, 1.0297053235191195e-05, 4.178533560711861e-05, 0.040122102 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 38.4 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.28, p95 0.88
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed7h 16m into the runtook 4m 26s
startup 89.6 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 89.6 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.9000363295763323e-05, 2.160383452747252e-05, 0.7278359510113522, 0.009419578015094555, 0.0010739839814404993, 0.003938763040954482, 0.000958080833962324, 9.751927820209515e-05, 7.093639708673459e-06, 4.1912010394140316e-05, 0.0398143 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 40.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.29, p95 0.93
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in, and the second frontier selfcheck passed on the final artifact, so /workspace/output is now being graded. The check covered startup, throughput, response format and training evidence. The architecture review was not part of it, and that review can still score the submission zero.
- Model: the final weights are the uniform average of four independently trained runs (
main2,v3,v3b,v3c; 13,600 steps in total), served with T=1. On my held-out dev tasks it scored about 0.716, against 0.704–0.716 for the single runs. That gap is within the noise of those tasks. - Selfcheck:
/healthanswered after 89.6 s against the 300 s limit. Throughput was 109.8 docs/s at concurrency 32 against the 50 required, with the 4,400 documents finishing in 40 s of the allowed 88 s. The check warns that held-out requests are about 1.75× larger, mostly in label length. My earlier stress test with 3× longer labels held at least 85 docs/s. - Manifest and evidence: I regenerated
training_manifest.json,architecture.jsonand the training config and logs for the four-run average. The manifest lists the base model revision, 80 datasets, 19 teachers and the two weight hashes. The weight hashes match the files inmodel/, and every evidence path exists.