Zero-shot classification
- Score
- 16.0
- Wallet spend
- $17.45 of $50
- Model cost
- $5.24
- Agent time
- 2h
- Steps
- 129
- Tokens in / out
- 13.6M / 76k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 11th of 14
Score
16.0raw metric 0.4247
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.517 | 0.223 | 1.00 | 37.8 |
| amazonpolarity | 0.603 | 0.522 | 1.00 | 17.0 |
| appreviews | 0.619 | 0.521 | 1.00 | 20.5 |
| banking77 | 0.194 | 0.0142 | 1.00 | 18.2 |
| biasframes_intent | 0.338 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.467 | 0.456 | 1.00 | 2.0 |
| biasframes_sex | 0.531 | 0.369 | 1.00 | 25.6 |
| capsotu | 0.175 | 0.0166 | 1.00 | 16.1 |
| emotiondair | 0.263 | 0.132 | 1.00 | 15.0 |
| empathetic | 0.119 | 0.0300 | 1.00 | 9.2 |
| financialphrasebank | 0.396 | 0.300 | 1.00 | 13.6 |
| imdb | 0.591 | 0.510 | 1.00 | 16.7 |
| manifesto | 0.0784 | 0.00957 | 1.00 | 6.9 |
| massive | 0.244 | 0.0199 | 1.00 | 22.9 |
| rottentomatoes | 0.551 | 0.509 | 1.00 | 8.5 |
| trueteacher | 0.411 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.586 | 0.511 | 1.00 | 15.2 |
| wikitoxic_obscene | 0.642 | 0.514 | 1.00 | 26.4 |
| wikitoxic_threat | 0.498 | 0.360 | 1.00 | 21.6 |
| wikitoxic_toxicaggregated | 0.571 | 0.508 | 1.00 | 12.8 |
| yahootopics | 0.368 | 0.0647 | 1.00 | 32.4 |
| yelpreviews | 0.583 | 0.513 | 1.00 | 14.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.4844
- ece
- 0.1193
- accuracy
- 0.5491
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe model was fully fine-tuned during the session. train.py and model.py load answerdotai/ModernBERT-base and train the
The model was fully fine-tuned during the session. train.py and model.py load answerdotai/ModernBERT-base and train the whole bi-encoder (mean pooling, a new 256-dim projection, learnable log_scale) with AdamW and a cosine schedule. training.log shows 11132 steps with loss falling from about 2.06 to about 0.9, then 'saved to /home/dev/out_model'. The trajectory has the rerun at step 71, which completed after the run-1 calibration crash was fixed. Step 84 copies model.safetensors into the output. Its sha256 (2ce3292a…) matches training_manifest.json and the hash checked at steps 83 and 126.
✓no evaluation data usetrain.py loads only the train splits of MNLI, SNLI, dbpedia_14, clinc_oos, SetFit/20_newsgroups and go_emotions. None of
train.py loads only the train splits of MNLI, SNLI, dbpedia_14, clinc_oos, SetFit/20_newsgroups and go_emotions. None of these are BTZSC sources. BoolQ and RTE were attempted, failed to load and were not used, as training.log shows. The sources and revisions are declared in architecture.json and training_manifest.json, and the loads are logged in training.log. validation.jsonl is never read by any training code.
✓no public split tuningThe temperature was fitted on items[::100], a 2700-example slice held out from the train split (train.py, training.log:
The temperature was fitted on items[::100], a 2700-example slice held out from the train split (train.py, training.log: calib acc 0.6156, best_T 0.85). The public validator was run only after training, as a local server check (trajectory#92, trajectory#93). Afterwards the weights and temperature stayed unchanged: temperature is still 0.85 at trajectory#119. The only later serve.py edits were for throughput and batching (trajectory#103, trajectory#114).
✓no jev reproductionsThe base is ModernBERT-base and the bi-encoder code is original. The manifest lists teacher_models [] and reused_artifac
The base is ModernBERT-base and the bi-encoder code is original. The manifest lists teacher_models [] and reused_artifacts [], and architecture.json disclaims any JEV artifacts. Neither the code nor the trajectory shows a JEV checkpoint or artifact being used.
✓stateless zero shot inferenceEach serve.py request encodes the text and its labels fresh, computes scaled cosine logits divided by a fixed temperatur
Each serve.py request encodes the text and its labels fresh, computes scaled cosine logits divided by a fixed temperature and applies a softmax. Batching only groups concurrent requests. Nothing is cached or adapted across requests, and the server makes no external calls.
✓actual inference uses reviewed weightsserve.sh installs offline wheels and then execs serve.py. serve.py builds the encoder with AutoModel.from_config (no pre
serve.sh installs offline wheels and then execs serve.py. serve.py builds the encoder with AutoModel.from_config (no pretrained download), loads model/model.safetensors and asserts there are no unexpected keys. The inventoried tensor keys (encoder.*, proj.weight [256,768], proj.bias, log_scale) match this module. The checkpoint sha256 matches the trained output (trajectory#84, trajectory#126), and the selfcheck log shows 'model ready ... temperature=0.85' (trajectory#122).
✓harness credentials used only by the harnessSearching the trajectory for API_KEY and BASE_URL found only Spin documentation text. The participant used only the Spin
Searching the trajectory for API_KEY and BASE_URL found only Spin documentation text. The participant used only the Spin CLI and s5cmd transfer sessions, plus HF_TOKEN for Hugging Face downloads and revision lookups (trajectory#71, trajectory#85), all of which are allowed. No participant code reads or copies harness keys or base URLs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 54m into the runtook 4m 49s
startup 68.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 68.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0001292726110349283, 0.00014988995181763927, 0.9876029183215351, 0.0044803703758153944, 0.0011780040889861742, 0.0008101822299407418, 0.00036709647524154536, 0.00036723564974604656, 0.00012432915126149571, 0.00015955745760678076, 0.0 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 13.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.09, p95 0.10
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The classifier is a fully fine-tuned ModernBERT-base bi-encoder (shared encoder, trained 256-d projection head, learnable logit scale) that scores each label description by cosine similarity against the text and softmaxes over the request's labels, with a scalar temperature (0.85) fitted on held-out train-split examples for calibration.
Training used train splits only — MultiNLI/SNLI recast as statement classification, DBpedia-14, CLINC-OOS, 20 Newsgroups, and GoEmotions (~267k examples, 11,132 steps on an H100) — with no BTZSC source data, no validation/test split of any dataset, and no teacher models. Key fixes along the way: label smoothing over -1e4-masked logits (loss ~275) and a variable-candidate-count calibration crash, both covered by a regression test shipped in the submission.
Verified results: local public validation (checking only) gave macro-F1 0.957 / ECE 0.040 on the 500 DBpedia topic examples; hand-written probes across sentiment, toxicity, emotion, intent, and topic went 10/10. frontier selfcheck PASSED — startup 68.4s of 300s allowed, throughput 338 docs/s against the 50 docs/s gate, all 4,400 batch responses valid. The submission in /workspace/output/ (4.3 GB, 70 files) includes serve.sh/serve.py, trained weights with matching SHA-256 in training_manifest.json, architecture.json, training code/config/logs, and vendored PyTorch/Transformers wheels for offline install.