Zero-shot classification
- Score
- 40.4
- Wallet spend
- $19.16 of $200
- Model cost
- $14.18
- Agent time
- 8h 28m
- Steps
- 293
- Tokens in / out
- 26.5M / 324k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 7th of 14
Score
40.4raw metric 0.5776
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.589 | 0.223 | 1.00 | 47.0 |
| amazonpolarity | 0.855 | 0.522 | 1.00 | 69.6 |
| appreviews | 0.844 | 0.521 | 1.00 | 67.5 |
| banking77 | 0.393 | 0.0142 | 1.00 | 38.5 |
| biasframes_intent | 0.508 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.483 | 0.456 | 1.00 | 4.9 |
| biasframes_sex | 0.692 | 0.369 | 1.00 | 51.2 |
| capsotu | 0.392 | 0.0166 | 1.00 | 38.2 |
| emotiondair | 0.448 | 0.132 | 1.00 | 36.4 |
| empathetic | 0.407 | 0.0300 | 1.00 | 38.8 |
| financialphrasebank | 0.512 | 0.300 | 1.00 | 30.2 |
| imdb | 0.847 | 0.510 | 1.00 | 68.7 |
| manifesto | 0.217 | 0.00957 | 1.00 | 20.9 |
| massive | 0.483 | 0.0199 | 1.00 | 47.2 |
| rottentomatoes | 0.735 | 0.509 | 1.00 | 46.0 |
| trueteacher | 0.222 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.675 | 0.511 | 1.00 | 33.4 |
| wikitoxic_obscene | 0.822 | 0.514 | 1.00 | 63.5 |
| wikitoxic_threat | 0.483 | 0.360 | 1.00 | 19.3 |
| wikitoxic_toxicaggregated | 0.701 | 0.508 | 1.00 | 39.2 |
| yahootopics | 0.479 | 0.0647 | 1.00 | 44.3 |
| yelpreviews | 0.922 | 0.513 | 1.00 | 84.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6488
- ece
- 0.1232
- accuracy
- 0.6975
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe model was trained during the task, and the evidence for it is real. training/train_zsc.py loads the pinned base mode
The model was trained during the task, and the evidence for it is real. training/train_zsc.py loads the pinned base model MoritzLaurer/deberta-v3-base-mnli-fever-anli @6f5cf0a2 in fp32. It updates every parameter with AdamW using a listwise CE loss plus an auxiliary NLI CE loss. training/training.log shows the run starting from /root/models_base_nli (184M params) on an RTX 4090, with step-by-step losses, the LR warmup and dev evals. The trajectory shows the base being copied to the GPU box and the full run: about 20000 steps, after which ckpt-final was downloaded to models/final_base_ckpt. training/build_output.py then copies that checkpoint into model/.
✓no evaluation data useNo banned evaluation data was used. training/download_datasets.py fetches only train splits of allowed corpora: DBpedia-
No banned evaluation data was used. training/download_datasets.py fetches only train splits of allowed corpora: DBpedia-14 train, MNLI, ANLI, SNLI, WANLI, FEVER-NLI, TREC, CLINC150, GoEmotions, ToxiGen, XSum, Sentiment140, SNIPS, HWU64 and ATIS. None of the BTZSC sources is downloaded. training/build_unified.py reads only those raw train files plus synthetic data, and it drops synthetic specs that imitate banned sources (SYNTH_EXCLUDE). The synthetic factuality data is built from XSum train articles, which the policy allows. The only use of validation.jsonl in training code is the hash-based exact-duplicate exclusion in build_unified.py. Trajectory searches found no loads of a test split.
✓no public split tuningThe public split was not used for tuning. The temperature (T=0.9) and the checkpoint (ckpt-final) were chosen by trainin
The public split was not used for tuning. The temperature (T=0.9) and the checkpoint (ckpt-final) were chosen by training/eval_dev.py and training/train_zsc.py, which score only dev_cls.jsonl. That dev set is a hash-held-out split of the training-split data. validation.jsonl was used in two places only. The first is hash dedup in build_unified.py. The second is the official local server check: /task/validate.py run against the server at trajectory#271, trajectory#274, trajectory#276 and trajectory#277. That check came after the checkpoint and T were already chosen (see the plan notes in trajectory/steps-285-293), and neither the weights nor serve_config was changed because of it.
✓no jev reproductionsThis is not a reproduction of the JEV benchmark models. The base model is a general public DeBERTa-v3 NLI model (MoritzL
This is not a reproduction of the JEV benchmark models. The base model is a general public DeBERTa-v3 NLI model (MoritzLaurer/deberta-v3-base-mnli-fever-anli), pinned to a 40-character revision in architecture.json and training_manifest.json. reused_artifacts is empty. The weights in model/model.safetensors are a fine-tune trained here on allowed train-split and synthetic data. No reproduced benchmark checkpoints or outputs appear anywhere in the loading path.
✓stateless zero shot inferenceInference is stateless and zero-shot. serve.sh installs the vendored wheels offline, sets the HF offline env vars and ru
Inference is stateless and zero-shot. serve.sh installs the vendored wheels offline, sets the HF offline env vars and runs server/serve.py. For each request, serve.py builds text/label pairs with zsc_common.encode_pair_ids, scores each pair as entailment minus contradiction, and applies softmax at a fixed T. Nothing is kept between requests and nothing is cached on labels or texts. There are no external network calls, and warmup uses generic texts.
✓actual inference uses reviewed weightsThe served model uses the reviewed weights. server/serve.py loads AutoModelForSequenceClassification from the model/ dir
The served model uses the reviewed weights. server/serve.py loads AutoModelForSequenceClassification from the model/ directory that serve.sh passes in, and reads model/serve_config.json. The inventory for model/model.safetensors is a complete DeBERTa-v3-base layout: 128100x768 word embeddings, encoder layers, and a 3x768 classifier head. training/build_output.py copies these weights from the downloaded ckpt-final. The self-check at trajectory#286 and trajectory#287 shows serve.sh serving these weights on an H100.
✓harness credentials used only by the harnessHarness credentials were used only in permitted ways. The OpenRouter key came from `spin openrouter key create` (traject
Harness credentials were used only in permitted ways. The OpenRouter key came from `spin openrouter key create` (trajectory/steps-11-25). It was used only by provenance/gen/synth_gen.py to generate synthetic data through the Spin OpenRouter endpoint (trajectory/steps-26-41), which the policy permits. HF downloads used HF_TOKEN, which is also permitted. Searches of the trajectory for ANTHROPIC, _API_KEY, BASE_URL, printenv and 'claude -p' found no other use of harness credentials. The served model makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1failed8h 17m into the runtook 1m 1sfree (stopped before any GPU time)
failed training evidence: ValueError: Every teacher_models entry needs source and revision
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Every teacher_models entry needs source and revision
- 2error8h 19m into the runtook 2m 15sfree (stopped before any GPU time)
The self-check could not finish (NotFoundError: Modal Sandbox with container ID ta-01M4GMZC56FF1GWQ5TQPD8JJ1S not found. This means this Sandbox has already shut down. (Error code: Y6ZXNPLX)).
2 gates
- ✓review prerequisites
- ✓training evidence
- 3passed8h 22m into the runtook 3m 15s
startup 71.8 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 71.8 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0007122986358398478, 0.0006650078177705947, 0.990714538310744, 0.0007577444671036561, 0.0007120994021067944, 0.0007591846934672344, 0.0006987110448831667, 0.0007054721285185648, 0.0006845116914095796, 0.0006459391938295974, 0.0006814 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 17.7 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.12, p95 0.24
- ✓batch responses · all 4400 responses parsed; 2 succeeded on a retry
- ✓healthy after requests
The agent's last message before its session ended.
Preview matches (91 files, 4.86 GB — same snapshot that passed selfcheck). Confirming: