Zero-shot classification
- Score
- 37.2
- Wallet spend
- $8.62 of $10
- Model cost
- $0.70
- Agent time
- 2h 2m
- Steps
- 319
- Tokens in / out
- 31.3M / 242k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 12th of 19
Score
37.2raw metric 0.5694
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.722 | 0.223 | 1.00 | 64.2 |
| amazonpolarity | 0.917 | 0.522 | 1.00 | 82.6 |
| appreviews | 0.835 | 0.521 | 1.00 | 65.5 |
| banking77 | 0.417 | 0.0142 | 1.00 | 40.9 |
| biasframes_intent | 0.524 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.457 | 0.456 | 1.00 | 0.1 |
| biasframes_sex | 0.510 | 0.369 | 1.00 | 22.3 |
| capsotu | 0.343 | 0.0166 | 1.00 | 33.2 |
| emotiondair | 0.480 | 0.132 | 1.00 | 40.1 |
| empathetic | 0.266 | 0.0300 | 1.00 | 24.4 |
| financialphrasebank | 0.573 | 0.300 | 1.00 | 39.0 |
| imdb | 0.892 | 0.510 | 1.00 | 78.0 |
| manifesto | 0.172 | 0.00957 | 1.00 | 16.4 |
| massive | 0.456 | 0.0199 | 1.00 | 44.5 |
| rottentomatoes | 0.773 | 0.509 | 1.00 | 53.8 |
| trueteacher | 0.423 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.671 | 0.511 | 1.00 | 32.6 |
| wikitoxic_obscene | 0.505 | 0.514 | 1.00 | 0.0 |
| wikitoxic_threat | 0.496 | 0.360 | 1.00 | 21.3 |
| wikitoxic_toxicaggregated | 0.636 | 0.508 | 1.00 | 26.0 |
| yahootopics | 0.551 | 0.0647 | 1.00 | 52.0 |
| yelpreviews | 0.908 | 0.513 | 1.00 | 81.2 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6191
- ece
- 0.0897
- accuracy
- 0.6632
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtraining/train_head.py trains a new ZSCHead (bilinear projection, cosine, pair MLP and learned temperature) on top of a
training/train_head.py trains a new ZSCHead (bilinear projection, cosine, pair MLP and learned temperature) on top of a frozen Qwen3-Embedding-0.6B. It uses AdamW, OneCycle and cross-entropy with random negatives. training/train.log records 5604 steps, with loss going from 7.31 to 0.53, and a config that matches model/head_config.json. training_manifest.json lists model/head.safetensors as the trained weight (sha256 8c2c7bd0...), and the trajectory recomputes that hash when the output is written. The head.safetensors inventory (wu/wv 512x1024, pair 1024x2048 -> 512x1024 -> 1x512, norms, log_temp) matches head_config (proj 512, hidden 1024). Nothing is reused (reused_artifacts is empty).
✓no evaluation data useThe training corpus is (a) synthetic text invented by teacher models through the Spin/OpenRouter proxy, made by training
The training corpus is (a) synthetic text invented by teacher models through the Spin/OpenRouter proxy, made by training/data/gen_synth.py from generic label pools (label names only, no evaluation texts), and (b) public TRAIN splits only, pulled with hf_hub_download in training/data/download_public.py. Those splits are 20newsgroups, clinc150, tweet_eval emotion/sentiment/irony/hate, go_emotions and tweet_sentiment_extraction. training/logs/download_public.log records each one with its dataset revision, row count and sha256. build_dataset.py skips synth_test files. No evaluation or test split appears in the data path.
✓no public split tuningEpoch selection (macro-F1) and temperature calibration (T=1.4 by NLL grid) in train_head.py use the sel/cal slices. buil
Epoch selection (macro-F1) and temperature calibration (T=1.4 by NLL grid) in train_head.py use the sel/cal slices. build_dataset.py carves these group-wise out of the training corpus (train.log: sel=1905, cal=1905). The trajectory's final summary says head J was chosen over the alternatives on this held-out training slice. Self-checks only confirmed that serving works; they report no scores and were not used for tuning.
✓no jev reproductionsThe only pretrained model shipped and served is Qwen/Qwen3-Embedding-0.6B at the declared revision. model/model.safetens
The only pretrained model shipped and served is Qwen/Qwen3-Embedding-0.6B at the declared revision. model/model.safetensors holds the Qwen3 encoder tensors, and the download log calls it a byte-identical upstream snapshot. The head is newly trained, and reused_artifacts is empty. Other models the participant downloaded during development were not shipped or used in serving.
✓stateless zero shot inferenceserver.py's classify() encodes each request's texts and its supplied candidate label descriptions per in-flight batch. I
server.py's classify() encodes each request's texts and its supplied candidate label descriptions per in-flight batch. It applies head(u, v)/temperature and returns a softmax over the supplied labels. It keeps no cache or state across requests, makes no external network calls (serve.sh sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE), and does no few-shot adaptation or grading-file access.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels (trusted binary audits approved), sets ZSC_MODEL_DIR to $HERE/model and execs serv
serve.sh installs the vendored wheels (trusted binary audits approved), sets ZSC_MODEL_DIR to $HERE/model and execs server.py. server.py loads AutoModel/AutoTokenizer from model/ with local_files_only and loads ZSCHead state from model/head.safetensors with strict load_state_dict, using the dims from model/head_config.json. Both weight inventories (model/model.safetensors and model/head.safetensors) are consistent with this call graph.
✓harness credentials used only by the harnessIn the trajectory the participant made its own Spin OpenRouter key with `spin openrouter key create` and used it as SPIN
In the trajectory the participant made its own Spin OpenRouter key with `spin openrouter key create` and used it as SPIN_KEY for teacher generation, which is the allowed proxy use. Spin compute/s5cmd moved files between the participant's machines. No use of harness credentials and no launches of harness agent workers were found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 17m into the runtook 1m 11sfree (stopped before any GPU time)
failed training evidence: ValueError: Every teacher_models entry needs source and revision
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Every teacher_models entry needs source and revision
- 2failed1h 21m into the runtook 6m 59s
failed startup: Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: rch.Size([256, 512]). size mismatch for pair.3.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([256]).
3 gates
- ✓review prerequisites
- ✓training evidence
- ✗startup · Server did not become healthy in 300s (last /health probe: HTTP 503 b'{"status": "loading"}'); end of its log: rch.Size([256, 512]). size mismatch for pair.3.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([256]). size mismatch for pair.6.weight: copying a param with shape torch.Size([1, 512]) from checkpoint, the shape in current model is torch.Size([1, 256]). Exception in thread loader: Traceback (most recent call last): File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner self.run() File "/usr/lib/python3.12/threading.py", line 1010, in run self._target(*self._args, **self._kwargs) File "/submission/server.py", line 175, in _loader self.model = Classifier() ^^^^^^^^^^^^ File "/submission/server.py", line 74, in __init__ self.head.load_state_dict(state) File "/tmp/zsc_site/torch/nn/modules/module.py", line 2624, in load_state_dict raise RuntimeError( RuntimeError: Error(s) in loading state_dict for ZSCHead: size mismatch for wu.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([384, 1024]). size mismatch for wv.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([384, 1024]). size mismatch for pair.0.weight: copying a param with shape torch.Size([1024, 2048]) from checkpoint, the shape in current model is torch.Size([512, 2048]). size mismatch for pair.0.bias: copying a param with shape torch.Size([1024]) from checkpoint, the shape in current model is torch.Size([512]). size mismatch for pair.3.weight: copying a param with shape torch.Size([512, 1024]) from checkpoint, the shape in current model is torch.Size([256, 512]). size mismatch for pair.3.bias: copying a param with shape torch.Size([512]) from checkpoint, the shape in current model is torch.Size([256]). size mismatch for pair.6 ...[clipped]
- 3passed1h 32m into the runtook 2m 56s
startup 63.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 63.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0008821853704008378, 0.0019670461616119028, 0.8066312198456792, 0.003982298190652203, 0.02881962531497284, 0.010256040161349901, 0.008797246179489605, 0.0013112233976736623, 5.1869632102020724e-05, 0.0010203655412268642, 0.0422895696 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 18.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.13, p95 0.15
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4passed1h 58m into the runtook 3m 9s
startup 63.8 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 63.8 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0023866108034300876, 0.0016065567841614325, 0.5361117210082026, 0.017441978380392523, 0.0459611185715589, 0.022971520682704003, 0.010525195683489105, 0.0033536616384835403, 0.0005286619622787904, 0.0019103559027534949, 0.259650493403 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 18.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.13, p95 0.16
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 21:39:57 UTC.
Final submission
Architecture — frozen Qwen/Qwen3-Embedding-0.6B (revision 97b0c61…, published weights, never fine-tuned) + a 3.68 M-parameter task-trained scoring head (bilinear projection on both sides + raw cosine + per-pair MLP over [u⊙v, |u−v|], divided by a learned temperature). Only the head is trained; the encoder is used exactly as published.
Training — 59,785 synthetic + public-corpus-train-split examples (20newsgroups, clinc150, tweet_eval×4, go_emotions, tweet_sentiment) with teacher-written texts and label descriptions; cross-entropy over the example's own label inventory plus 48 random negative descriptions, OneCycleLR, 6 epochs, epoch chosen by macro-F1 on a held-out selection slice, one scalar temperature fitted on an independent calibration half. No validation/test split of any public dataset and none of the 500 local rows were used for training, calibration or selection.
Last change this session — description style turned out to be a real generalisation axis, so the corpus was augmented: half the training rows have their whole label inventory rewritten in a terse style (bare label name). On the same held-out slice scored three ways (all-sentence descriptions / all-terse / 50-50 mix), head J (row-level 50 % terse) scored 0.603 / 0.574 / 0.582 versus 0.595 / 0.482 / 0.528 for the un-augmented head — better on every style mix, while a 10-epoch schedule (head K) and a two-head ensemble both measured worse and were discarded.
Verification — final frontier selfcheck PASSED on an H100: startup 63.8 s (limit 300), throughput 241.0 docs/s (limit 50), all 4,400 responses parsed, correct {"probabilities": […]} contract, 87 files / 5.18 GB with no symlinks and no packaged corpora.