Zero-shot classification
- Score
- 0.0
- Wallet spend
- $14.40 of $50
- Model cost
- $2.37
- Agent time
- 8h 42m
- Steps
- 893
- Tokens in / out
- 88.5M / 629k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 15th of 19
Trusted throughput client failed: TimeoutError: Whole-batch deadline exceeded
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe model starts from answerdotai/ModernBERT-base at the pinned revision. common.py's build_model loads the encoder stri
The model starts from answerdotai/ModernBERT-base at the pinned revision. common.py's build_model loads the encoder strictly and adds a new nn.Linear(768,1) head. Training then ran in 4 stages during the session (stage1 → zs3 → zs4 → zs5), each with its own log (logs/train_stage1.log, train3/4/5.log). The safetensors inventory matches PairScorer: encoder.* layers plus head.weight [1,768] and head.bias [1]. In trajectory step 881 the shipped digest f8f8fa6d… is verified against the manifest and traced to models/zs5.
✓no evaluation data useAll training corpora are declared train splits at pinned revisions: dbpedia_14, 20 newsgroups, CLINC-150 train (texts al
All training corpora are declared train splits at pinned revisions: dbpedia_14, 20 newsgroups, CLINC-150 train (texts also found in val, test or oos are dropped), tweet_eval, tweet_sentiment_extraction, measuring-hate-speech, go_emotions and XSum train for synthetic faithfulness corruptions. The rest is synthetic data written by teacher LLMs through the Spin proxy. download_real.py uses train-* allow_patterns, and data_manifest.json records the download commands and logs. None of the data comes from evaluation or test splits.
✓no public split tuningcalibrate.py and eval_server.py choose the temperature and the checkpoint on a dev split carved from the training corpor
calibrate.py and eval_server.py choose the temperature and the checkpoint on a dev split carved from the training corpora by train.py's split_tasks, using leave-one-out over the real dev tasks. Step 882 shows zs5 was picked by dev-split leave-one-out. The public validation file was run through validate.py only afterwards, as a server check (step 880). prepare_real.py and add_sentence_variants.py read the validation label strings only to remove generated descriptions that exactly match them. That is a decontamination ban-list, not training, calibration or selection on those rows.
✓no jev reproductionstraining_manifest.json lists reused_artifacts as empty. The only pretrained component is ModernBERT-base. The weights we
training_manifest.json lists reused_artifacts as empty. The only pretrained component is ModernBERT-base. The weights were trained in this session from that base plus a new head. I found no reproduction or reuse of judge or evaluation artifacts.
✓stateless zero shot inferenceserver.py tokenises each request's text and label descriptions, scores each pair with the cross-encoder, and applies a s
server.py tokenises each request's text and label descriptions, scores each pair with the cross-encoder, and applies a softmax over that request's labels divided by a fixed temperature (0.95). Dynamic batching shares no state between requests. There is no cache, retrieval, learning at request time or external call. Unseen labels are handled zero-shot through their descriptions.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1, finds the directory holding model.safetensors and zsc_config.json, and execs server.py.
serve.sh sets HF_HUB_OFFLINE=1, finds the directory holding model.safetensors and zsc_config.json, and execs server.py. server.py calls common.load_model, which does load_file(model/model.safetensors) and a strict load_state_dict, then reads the temperature from model/zsc_config.json. The inventory matches the PairScorer structure. The wheels are approved in the trusted binary audits. Serving makes no remote model calls.
✓harness credentials used only by the harnessThe trajectory shows no use of ANTHROPIC or OPENROUTER environment keys and no extra claude CLI sessions. Teacher LLM ca
The trajectory shows no use of ANTHROPIC or OPENROUTER environment keys and no extra claude CLI sessions. Teacher LLM calls in gen_synth.py and prepare_real.py go through the Spin OpenRouter proxy, using SPIN_OR_BASE and a key from `spin openrouter key create`, which is allowed. HF downloads use the permitted HF_TOKEN path.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 33m into the runtook 52sfree (stopped before any GPU time)
failed training evidence: ValueError: Training evidence must declare positive parameter-update steps
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Training evidence must declare positive parameter-update steps
- 2passed1h 36m into the runtook 2m 41s
startup 30.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 30.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0714166132, 0.0714166132, 0.0714445154, 0.0714166132, 0.0714166132, 0.0714445158, 0.0714445158, 0.0714166132, 0.0714166132, 0.0714445158, 0.0714166132, 0.0714166132, 0.0714445158, 0.0714445158]}
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 46.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.33, p95 1.37
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed5h 11m into the runtook 2m 48s
startup 29.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 29.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.25938e-05, 6.9722e-05, 0.9307692463, 0.0150669755, 0.0001767217, 0.0094638146, 4.46158e-05, 6.47227e-05, 1.00744e-05, 6.4467e-06, 0.0329093002, 0.0109823019, 2.28384e-05, 0.000400626]}
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 39.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.28, p95 1.26
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4failed6h 23m into the runtook 3m 49s
startup 32.4 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 32.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.9391e-06, 1.29929e-05, 0.9217338877, 0.0026001239, 4.41339e-05, 0.005246451, 7.9519e-05, 1.95311e-05, 1.3497e-06, 1.6178e-06, 0.0029785263, 0.0670277153, 5.4953e-06, 0.000246717]}
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 5passed7h 37m into the runtook 3m 6s
startup 30.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 30.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [5.1674e-06, 6.00338e-05, 0.9336030054, 0.0076360497, 3.45052e-05, 0.0127702841, 4.92605e-05, 0.0001377725, 4.7744e-06, 4.0757e-06, 0.0343308707, 0.0109014202, 1.13989e-05, 0.0004513815]}
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 48.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.35, p95 1.47
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final artifact — /workspace/output (118 files, 4.41 GB)
| Model | ModernBERT-base cross-encoder, [CLS] description [SEP] text [SEP] → scalar, softmax over one temperature (0.95) |
| Weights | 4-stage training run (2 epochs + 3 continuations), model/model.safetensors sha256 f8f8fa6d… |
| Serving | serve.sh → offline wheel unpack into /tmp, ThreadingHTTPServer + dynamic batching, bf16 |
| Gates | startup 30.2 s / 300 s · throughput 91.6 docs/s (88 % to spare) · last selfcheck PASSED |
| Public check | 0.9571 macro-F1·(1−ECE), 0 failed requests |
How it was selected. Only the dev split (carved out of the training corpora + teacher-written synthetic tasks) was used for training, calibration and model choice; the 500 public rows were only ever used to check that the server answers. Each epoch was added only if it improved the leave-one-out score over the 12 real dev tasks: 0.6984 (2 ep) → 0.7147 (3) → 0.7303 (4) → 0.7722 (5, shipped), the last winning 29 of 36 dev datasets (+0.0397 ± 0.0072 paired). The serving temperature is the winner of a leave-one-out comparison between five calibration families; a margin calibration and an isotonic map were tried and rejected.
Two things worth flagging.
- A 384-token truncation budget scored higher on the dev split (+0.19 on the one long-document task) but failed the grading throughput gate at 26 docs/s, so the trained 128-token budget was kept — recorded in
architecture.jsonandlogs/. - The last selfcheck ran against the 4th-epoch weights; the 5th epoch finished afterwards and was swapped in. The serving path it measured is unchanged (same code, architecture, budget and batching), the new weights were smoke-tested on the leased GPU, and the substitution is documented in
logs/selfcheck4.logand in the manifest'sselectionfield.
Training used only train splits of non-excluded corpora (DBpedia-14, 20 Newsgroups, CLINC-150, tweet_eval, tweet sentiment, hate speech, GoEmotions, XSum) plus 1681 teacher-written synthetic tasks, with no excluded source in any split; provenance, revisions and generation code are in training_manifest.json and data_manifest.json.