Zero-shot classification
- Score
- 37.7
- Wallet spend
- $11.41 of $200
- Model cost
- $18.17
- Agent time
- 2h 50m
- Steps
- 319
- Tokens in / out
- — / —
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 10th of 19
Score
37.7raw metric 0.5778
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.800 | 0.223 | 1.00 | 74.2 |
| amazonpolarity | 0.805 | 0.522 | 1.00 | 59.2 |
| appreviews | 0.815 | 0.521 | 1.00 | 61.3 |
| banking77 | 0.494 | 0.0142 | 1.00 | 48.6 |
| biasframes_intent | 0.349 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.440 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.643 | 0.369 | 1.00 | 43.4 |
| capsotu | 0.489 | 0.0166 | 1.00 | 48.0 |
| emotiondair | 0.468 | 0.132 | 1.00 | 38.6 |
| empathetic | 0.379 | 0.0300 | 1.00 | 36.0 |
| financialphrasebank | 0.610 | 0.300 | 1.00 | 44.3 |
| imdb | 0.755 | 0.510 | 1.00 | 50.0 |
| manifesto | 0.244 | 0.00957 | 1.00 | 23.7 |
| massive | 0.529 | 0.0199 | 1.00 | 51.9 |
| rottentomatoes | 0.749 | 0.509 | 1.00 | 48.9 |
| trueteacher | 0.400 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.659 | 0.511 | 1.00 | 30.3 |
| wikitoxic_obscene | 0.538 | 0.514 | 1.00 | 5.0 |
| wikitoxic_threat | 0.603 | 0.360 | 1.00 | 38.0 |
| wikitoxic_toxicaggregated | 0.601 | 0.508 | 1.00 | 19.0 |
| yahootopics | 0.550 | 0.0647 | 1.00 | 51.9 |
| yelpreviews | 0.792 | 0.513 | 1.00 | 57.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6789
- ece
- 0.1440
- accuracy
- 0.7202
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe classifier was trained during the task. train.py fine-tunes three models from pinned public bases: a RoBERTa-large b
The classifier was trained during the task. train.py fine-tunes three models from pinned public bases: a RoBERTa-large bi-encoder with a set head, a DeBERTa-v3-large cross-encoder and a RoBERTa-large cross-encoder. The bases are microsoft/deberta-v3-large@64a8c8ea and FacebookAI/roberta-large@722cf37b. Training uses NLI warmup and then episodic training. train.py then fits the Stage-3 SetAwareCalibrator on calib_split. The weight inventories show full trained encoder tensors plus new heads (cls_head, alpha_set, alpha_nli, and the calibrator MLP and bin offsets). The hashes in training_manifest.json match the trajectory's training and recalibration outputs (trajectory#266, #270) and the final manifest check (trajectory#304). reused_artifacts is empty.
✓no evaluation data useNo BTZSC source appears in the training data. prepare_data.py loads only split='train' at pinned revisions from: clinc_o
No BTZSC source appears in the training data. prepare_data.py loads only split='train' at pinned revisions from: clinc_oos, Bitext customer support, go_emotions, SetFit/20_newsgroups, news-category, lex_glue ledgar and scotus, tweet_eval:sentiment, xsum, WANLI and MNLI. All of these are declared in architecture.json, training_manifest.json and data_provenance.json. None is on the DATA_POLICY exclusion list (AG News, Amazon, App Reviews, Banking77, SBF, CAP/SOTU, DAIR Emotion, EmpatheticDialogues, Financial PhraseBank, IMDb, Manifesto, MASSIVE, Rotten Tomatoes, TrueTeacher, Jigsaw, Yahoo, Yelp). Searches for these sources in code and trajectory found no use. The synthetic data was generated fresh by gemini-2.5-flash from instructions, not from excluded examples. The calib and dev_eval splits are carved from these train splits.
✓no public split tuningvalidation.jsonl (500 DBpedia rows) was used only for server, throughput and timing checks. Choices that affect results
validation.jsonl (500 DBpedia rows) was used only for server, throughput and timing checks. Choices that affect results were made on train-derived splits: - The top-M sweep was picked on dev_eval (trajectory#254, #255), and the final 5/4 setting came from a dev_eval-only sweep (trajectory#292-#295). - The calibrator and fusion presets were fitted and selected on calib_split (train.py, trajectory#264-#270). Public scores were printed alongside but never drove a choice. They fell from 0.8755 to about 0.842 without any revert (trajectory#258, #301). No source file references validation.jsonl.
✓no jev reproductionsThe only bases are the official microsoft/deberta-v3-large and FacebookAI/roberta-large at pinned revisions. The teacher
The only bases are the official microsoft/deberta-v3-large and FacebookAI/roberta-large at pinned revisions. The teacher is google/gemini-2.5-flash. No Jeff, Laya, Open-Jev, JEV or typesafe artifacts appear anywhere in the code, manifests or trajectory (searches returned nothing).
✓stateless zero shot inferenceEach request in server.py runs independently. The text and the supplied labels are encoded fresh, labels are permuted an
Each request in server.py runs independently. The text and the supplied labels are encoded fresh, labels are permuted and the original order restored, and the model is evaluated with fixed loaded weights. The code explicitly avoids caching label embeddings across requests. It has no online learning, no persistence across requests and no external network calls, and serve.sh sets offline mode. The selfcheck passed with valid responses (trajectory#313).
✓actual inference uses reviewed weightsserver.py builds the backbones from the shipped tokenizer configs with AutoModel.from_config, so no pretrained download
server.py builds the backbones from the shipped tokenizer configs with AutoModel.from_config, so no pretrained download is involved. It then calls load_state_dict on weights/bi_roberta, ce_deberta, ce_roberta and calibrator.safetensors, which are the inventoried files. Their sha256 values match training_manifest.json, as verified in trajectory#304. serve.sh runs `exec python3 server.py`. The shipped training_config.json (top_m 5/4) matches the final edit in trajectory#295.
✓harness credentials used only by the harnessThe teacher was reached through the harness's own channel. The OpenRouter key was created with `spin openrouter key crea
The teacher was reached through the harness's own channel. The OpenRouter key was created with `spin openrouter key create` into /tmp/or_key.json and used only by generate_synthetic_data.py for gemini-2.5-flash teacher calls through the Spin-budgeted endpoint, which is permitted. No raw credentials are embedded in shipped files. The serving path runs offline and uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed2h 43m into the runtook 3m 26s
startup 77.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 77.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.01588666990487679, 0.013744632747568784, 0.587735851852671, 0.048435694751869984, 0.015451887233920988, 0.01551758632952028, 0.013614622393336188, 0.02529705594696838, 0.014176857744629356, 0.013530557547600668, 0.1516860873154019, 0 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 37.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.27, p95 0.36
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests