Zero-shot classification
- Score
- 43.2
- Wallet spend
- $9.82 of $200
- Model cost
- $6.95
- Agent time
- 1h 58m
- Steps
- 166
- Tokens in / out
- 29.6M / 195k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
43.2raw metric 0.6023
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.754 | 0.223 | 1.00 | 68.4 |
| amazonpolarity | 0.852 | 0.522 | 1.00 | 68.9 |
| appreviews | 0.858 | 0.521 | 1.00 | 70.2 |
| banking77 | 0.409 | 0.0142 | 1.00 | 40.0 |
| biasframes_intent | 0.659 | 0.527 | 1.00 | 27.8 |
| biasframes_offensive | 0.651 | 0.456 | 1.00 | 35.8 |
| biasframes_sex | 0.501 | 0.369 | 1.00 | 20.8 |
| capsotu | 0.521 | 0.0166 | 1.00 | 51.2 |
| emotiondair | 0.484 | 0.132 | 1.00 | 40.5 |
| empathetic | 0.168 | 0.0300 | 1.00 | 14.2 |
| financialphrasebank | 0.465 | 0.300 | 1.00 | 23.6 |
| imdb | 0.764 | 0.510 | 1.00 | 51.9 |
| manifesto | 0.153 | 0.00957 | 1.00 | 14.5 |
| massive | 0.704 | 0.0199 | 1.00 | 69.8 |
| rottentomatoes | 0.757 | 0.509 | 1.00 | 50.4 |
| trueteacher | 0.393 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.776 | 0.511 | 1.00 | 54.2 |
| wikitoxic_obscene | 0.787 | 0.514 | 1.00 | 56.1 |
| wikitoxic_threat | 0.552 | 0.360 | 1.00 | 30.1 |
| wikitoxic_toxicaggregated | 0.758 | 0.508 | 1.00 | 50.8 |
| yahootopics | 0.416 | 0.0647 | 1.00 | 37.5 |
| yelpreviews | 0.872 | 0.513 | 1.00 | 73.7 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6688
- ece
- 0.1033
- accuracy
- 0.7130
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain/train.py fine-tunes the full intfloat/e5-base-v2 backbone at the pinned revision f52bf8ec, plus a learned logit sc
train/train.py fine-tunes the full intfloat/e5-base-v2 backbone at the pinned revision f52bf8ec, plus a learned logit scale. It uses AdamW and an InfoNCE-style cross-entropy loss over label descriptions. Training ran in three chained stages: A (8000 steps), A2 (6000) and B (8000), with B resuming from outA2/best. The training logs show real step, loss and eval lines with timestamps inside the session (for example training_B.log starts at 08:26:30 and resumes from /root/zsc/outA2/best). The trajectory shows these runs on the Spin GPU and the outB/best checkpoint being transferred into the submission.
✓no evaluation data useAll real corpora are loaded only from train splits (prep_data.py, prep_data2.py, prep_data3.py): - GLUE mnli/qnli/rte/mr
All real corpora are loaded only from train splits (prep_data.py, prep_data2.py, prep_data3.py): - GLUE mnli/qnli/rte/mrpc/qqp, SNLI, ANLI train rounds, DBpedia-14 train - 20NG, HuffPost, GoEmotions, tweet_eval train, CLINC150, HWU64, ATIS, Davidson, OLID - XSum train for consistency pairs None of these is a BTZSC source or one of the named derivatives (SST/MR/subj, CNN/DM, FactCC, FRANK, SummEval, Jigsaw, MASSIVE, SBF, etc.). Davidson and HWU64 are upstream corpora, not named derivatives. The synthetic families from gen_synth.py are generated by deepseek-chat from original label wordings and style guides only, with no real source examples. Every source is declared with revisions in dataset_manifest.json and training_manifest.json; downloads are recorded in the prep logs.
✓no public split tuningDev selection uses held-out slices of training data (split_dev in prep_data.py; evaluate() in train.py). validation.json
Dev selection uses held-out slices of training data (split_dev in prep_data.py; evaluate() in train.py). validation.jsonl (DBpedia test rows) was touched only by /task/validate.py, as a final protocol check at steps 152-153. The later stage C experiment was rejected by comparing dev means (0.8035 vs 0.8061), not by its validation score. train/loadgen.py explicitly uses only data/*.dev.jsonl train slices.
✓no jev reproductionsThe initialization is the ordinary pretrained encoder intfloat/e5-base-v2 at a pinned revision, as declared in architect
The initialization is the ordinary pretrained encoder intfloat/e5-base-v2 at a pinned revision, as declared in architecture.json and train.py. The weight inventory is a standard BERT-base layout (30522x768 embeddings). reused_artifacts is empty. No Jeff, Laya, Open-Jev or other JEV-reproduction model or data appears in the code, manifests or trajectory; the only mention of 'jeff' is in the policy text.
✓stateless zero shot inferenceserver.py encodes each request's text and labels freshly with the model, using mean pooling and L2 normalization. It ret
server.py encodes each request's text and labels freshly with the model, using mean pooling and L2 normalization. It returns softmax(scale * cosine similarity) over that request's labels. There is no caching of label embeddings, no state carried across requests, no dataset-specific logic and no external calls. Dynamic batching of concurrent independent requests is allowed by the contract.
✓actual inference uses reviewed weightsserve.sh runs server.py offline (HF_HUB_OFFLINE=1). server.py loads AutoModel/AutoTokenizer from the local model/ direct
serve.sh runs server.py offline (HF_HUB_OFFLINE=1). server.py loads AutoModel/AutoTokenizer from the local model/ directory with local_files_only=True and trust_remote_code=False, and reads the logit scale from model/scale.json. The model/model.safetensors inventory (437951328 bytes, BERT-base tensors) matches the hash listed in training_manifest.json. The trajectory shows the file being transferred from outB/best (steps 142-145) and backed up unchanged at step 159. Stage C was not installed. All installed wheels have approved trusted audits.
✓harness credentials used only by the harnessThe OPENROUTER_KEY and OPENROUTER_BASE_URL used by gen_synth.py came from `spin openrouter key create` (trajectory step
The OPENROUTER_KEY and OPENROUTER_BASE_URL used by gen_synth.py came from `spin openrouter key create` (trajectory step 42): a Spin-scoped key with base_url api.spin.inductionlabs.com, which is the allowed Spin OpenRouter proxy. The storage session used for transfer was a Spin s5cmd session export. HF_TOKEN was used only for downloads. No harness credentials or base URLs were used, and no additional claude/codex agent CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 44m into the runtook 3m 1s
startup 83.9 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 83.9 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.005061764517559332, 0.005712502915325018, 0.9346457580746172, 0.006427261361322572, 0.004393515857550858, 0.004285498731759082, 0.004298748658675984, 0.0049406674312288314, 0.004237579388984403, 0.005814293683739769, 0.00644698537743 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 10.4 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.08, p95 0.12
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Result
Zero-shot text classifier served via POST /classify, submitted in /workspace/output/ and verified with frontier selfcheck: PASSED on H100 — startup 83.9s (72% spare), throughput 422 docs/s (744% spare), all 4400 responses parsed.
Approach
Model: bi-encoder (E5-base backbone, mean pooling, L2 norm, learned logit scale), full fine-tuned 22k steps in 3 stages on 36 datasets / 716k rows. Per step: one dataset sampled (sqrt weighting), full label set with randomly varied description styles plus cross-dataset distractors (up to 256 candidates), cross-entropy with label smoothing.
Training data (train splits / single-split corpora only; no val/test anywhere): MNLI, SNLI, ANLI, QNLI, RTE, MRPC, QQP, DBpedia, 20 Newsgroups, HuffPost, GoEmotions, TweetEval ×5, CLINC150, HWU64, ATIS, Davidson, OLID, rule-built XSum consistency pairs, and 43k synthetic examples (reviews, CAP/Manifesto topics, assistant/banking intents, finance, dialogue emotion, toxicity framings) generated with deepseek-chat. No BTZSC source in any split, no JEV-reproduction code or weights. Selection used only held-out train slices (dev mean 0.806).
Serving: stdlib HTTP server with dynamic GPU batching, bf16, no cross-request caching; offline install from 49 vendored PyPI wheels (torch 2.8 cu128). Local validate.py protocol check exited 0 (run after freezing the model; nothing was selected on it).
Files
serve.sh, server.py, model/ (weights, tokenizer, scale), wheels/, train/ (training/prep/synth code, configs, logs, data manifests), training_manifest.json, architecture.json.