Zero-shot classification
- Score
- 62.1
- Wallet spend
- $47.64 of $200
- Model cost
- $76.74
- Agent time
- 5h
- Steps
- 182
- Tokens in / out
- 36.7M / 225k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 1st of 14
Score
62.1raw metric 0.7285
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.877 | 0.223 | 1.00 | 84.2 |
| amazonpolarity | 0.941 | 0.522 | 1.00 | 87.7 |
| appreviews | 0.894 | 0.521 | 1.00 | 77.9 |
| banking77 | 0.640 | 0.0142 | 1.00 | 63.5 |
| biasframes_intent | 0.712 | 0.527 | 1.00 | 39.2 |
| biasframes_offensive | 0.697 | 0.456 | 1.00 | 44.3 |
| biasframes_sex | 0.640 | 0.369 | 1.00 | 42.8 |
| capsotu | 0.638 | 0.0166 | 1.00 | 63.2 |
| emotiondair | 0.399 | 0.132 | 1.00 | 30.7 |
| empathetic | 0.459 | 0.0300 | 1.00 | 44.3 |
| financialphrasebank | 0.790 | 0.300 | 1.00 | 70.0 |
| imdb | 0.937 | 0.510 | 1.00 | 87.2 |
| manifesto | 0.346 | 0.00957 | 1.00 | 34.0 |
| massive | 0.661 | 0.0199 | 1.00 | 65.4 |
| rottentomatoes | 0.856 | 0.509 | 1.00 | 70.6 |
| trueteacher | 0.507 | 0.433 | 1.00 | 13.0 |
| wikitoxic_insult | 0.874 | 0.511 | 1.00 | 74.3 |
| wikitoxic_obscene | 0.881 | 0.514 | 1.00 | 75.6 |
| wikitoxic_threat | 0.921 | 0.360 | 1.00 | 87.7 |
| wikitoxic_toxicaggregated | 0.842 | 0.508 | 1.00 | 67.8 |
| yahootopics | 0.567 | 0.0647 | 1.00 | 53.8 |
| yelpreviews | 0.945 | 0.513 | 1.00 | 88.7 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7793
- ece
- 0.0737
- accuracy
- 0.8073
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe weights were trained during the session. Run1 did 1393 steps of LoRA (r=32) plus MLP label-head training. Run2 did 1
The weights were trained during the session. Run1 did 1393 steps of LoRA (r=32) plus MLP label-head training. Run2 did 1499 steps continuing from run1. training/train.py did the training and export.py merged the LoRA into the served safetensors. The manifest lists the trained weights (two model shards plus head.pt), and reused_artifacts is empty. The trajectory shows both runs, the export and the temperature fit happening in the session.
✓no evaluation data useAll real training data comes from public HF train splits loaded with split="train" in build_real.py and build_real2.py.
All real training data comes from public HF train splits loaded with split="train" in build_real.py and build_real2.py. BTZSC and DBpedia are explicitly excluded. Datasets that might overlap evaluation either raise errors or are never called. hwu64 failed and contributed 0 rows, per build_real.log and build_real2.log. Synthetic data was generated from hand-written domain descriptions, with no evaluation examples (synth_gen.py). Factual pairs use XSum train articles (synth_factual.py); 'test_xsum.jsonl' is only an upload filename for that train-split data. Every source is declared in the manifest with sha256 digests. Where public sources have no retained download command or log, they are recorded as unverified.
✓no public split tuningThe softmax temperature (T=1.1) was grid-searched by fit_temp.py on held-out tasks drawn from train splits (build_mix.py
The softmax temperature (T=1.1) was grid-searched by fit_temp.py on held-out tasks drawn from train splits (build_mix.py HELD_REAL). validation.jsonl was used only for post-hoc smoke and stress tests, scored after each export (trajectory#103 and step 172). Run2 was a pre-planned continuation started immediately afterward (trajectory#104). Run2 was shipped even though its public score was slightly lower than run1's (0.9719 vs 0.975), so nothing was selected or tuned on the public split.
✓no jev reproductionsThe base model is Qwen/Qwen3-4B-Instruct-2507 at a pinned revision, fine-tuned with the in-session LoRA. architecture.js
The base model is Qwen/Qwen3-4B-Instruct-2507 at a pinned revision, fine-tuned with the in-session LoRA. architecture.json and the data-policy notes state no JEV use, and no JEV checkpoints or reproductions appear in the code, manifest or trajectory.
✓stateless zero shot inferenceserver_mp.py scores each request independently. It builds the prompt from that request's text and labels (zsc_model.Prom
server_mp.py scores each request independently. It builds the prompt from that request's text and labels (zsc_model.PromptBuilder), runs a backbone forward pass, applies the label head to the hidden states at the label positions, divides by the fixed T and applies a per-request softmax. Dynamic batching does not mix information across requests. There is no response caching, no learning from requests and no network access (HF_HUB_OFFLINE=1).
✓actual inference uses reviewed weightsserve.sh sets ZSC_MODEL_DIR=$HERE/model and runs server_mp.py. server_mp.py loads AutoModelForCausalLM.from_pretrained(M
serve.sh sets ZSC_MODEL_DIR=$HERE/model and runs server_mp.py. server_mp.py loads AutoModelForCausalLM.from_pretrained(MODEL_DIR) and head.load_state_dict(torch.load(model/head.pt)). The weight inventories show standard Qwen3-4B BF16 tensors in the two shards and LabelHead-shaped tensors in head.pt. The only uninspected file is a Triton kernel cache .so (cuda_utils), which does not hold model weights.
✓harness credentials used only by the harnessThe trajectory shows no use of harness credentials. The only API key the participant used is their own spin agent-wallet
The trajectory shows no use of harness credentials. The only API key the participant used is their own spin agent-wallet OpenRouter key in or_key.json, which synth_gen.py and synth_factual.py use for teacher generation, as the task setup allows (trajectory#6, trajectory#22). The session contains no ANTHROPIC or other harness credential use.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 58m into the runtook 4m 49s
startup 67.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 67.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.1585127257380906e-05, 4.184182840011781e-06, 0.9630783137589781, 0.0012231893233652735, 1.3550082240711165e-05, 0.0004266615141327331, 1.6358787224596977e-05, 3.1497538776663504e-06, 1.865868498207139e-06, 6.32701405646441e-06, 0.001 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 44.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.31, p95 1.16
- ✓batch responses · all 4400 responses parsed; 4 succeeded on a retry
- ✓healthy after requests
- 2passed3h 48m into the runtook 5m 21s
startup 122.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 122.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.1585127299112654e-05, 4.184182905402719e-06, 0.9630783128241106, 0.0012231894093249602, 1.3550082581977561e-05, 0.0004266615272079137, 1.6358787603481838e-05, 3.1497540367902873e-06, 1.8658684874397468e-06, 6.3270143661738005e-06, 0. ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 28.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.21, p95 0.28
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed4h 50m into the runtook 6m 2s
startup 138.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 138.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.4478720611432702e-05, 4.1572328923904645e-06, 0.8689877209411365, 0.004427139089202253, 1.059663805491899e-05, 0.0010439837133450964, 5.956230837231373e-05, 4.908405469277907e-06, 2.554883341275591e-06, 1.3583131192855655e-05, 0.0035 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 31.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.23, p95 0.32
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
All checks are done, so the one remaining item is the final submission.