Zero-shot classification
- Score
- 46.7
- Wallet spend
- $13.96 of $200
- Model cost
- $18.69
- Agent time
- 46m
- Steps
- 98
- Tokens in / out
- 11.6M / 97k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 5th of 14
Score
46.7raw metric 0.6226
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.820 | 0.223 | 1.00 | 76.8 |
| amazonpolarity | 0.913 | 0.522 | 1.00 | 81.8 |
| appreviews | 0.866 | 0.521 | 1.00 | 71.9 |
| banking77 | 0.444 | 0.0142 | 1.00 | 43.6 |
| biasframes_intent | 0.667 | 0.527 | 1.00 | 29.6 |
| biasframes_offensive | 0.563 | 0.456 | 1.00 | 19.6 |
| biasframes_sex | 0.645 | 0.369 | 1.00 | 43.7 |
| capsotu | 0.469 | 0.0166 | 1.00 | 46.0 |
| emotiondair | 0.446 | 0.132 | 1.00 | 36.1 |
| empathetic | 0.383 | 0.0300 | 1.00 | 36.4 |
| financialphrasebank | 0.611 | 0.300 | 1.00 | 44.4 |
| imdb | 0.916 | 0.510 | 1.00 | 82.8 |
| manifesto | 0.199 | 0.00957 | 1.00 | 19.2 |
| massive | 0.477 | 0.0199 | 1.00 | 46.7 |
| rottentomatoes | 0.854 | 0.509 | 1.00 | 70.2 |
| trueteacher | 0.342 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.666 | 0.511 | 1.00 | 31.6 |
| wikitoxic_obscene | 0.784 | 0.514 | 1.00 | 55.5 |
| wikitoxic_threat | 0.558 | 0.360 | 1.00 | 31.0 |
| wikitoxic_toxicaggregated | 0.631 | 0.508 | 1.00 | 25.1 |
| yahootopics | 0.500 | 0.0647 | 1.00 | 46.6 |
| yelpreviews | 0.944 | 0.513 | 1.00 | 88.4 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6853
- ece
- 0.1014
- accuracy
- 0.7261
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the task. extract_features.py computes frozen DeBERTa CLS features. train.py then trains a 1024
Training happened during the task. extract_features.py computes frozen DeBERTa CLS features. train.py then trains a 1024-dim listwise head with Adam, starting from the entailment row of classifier.weight and using an L2 anchor penalty. calibrate.py fits 3 temperature coefficients. Both ran on the leased H100: trajectory#47 runs extract_features followed by train.py, and trajectory#74-75 show the calibration steps. The weight inventories confirm what this produced: head.safetensors holds weight [1024] F32 and calibration.safetensors holds coefficients [3]. training_manifest.json records 6000 steps and the head and calibration hashes.
✓no evaluation data useNo banned data is used. The declared training sources are MoritzLaurer/synthetic_zeroshot_mixtral_v0.1, clinc_oos, go_em
No banned data is used. The declared training sources are MoritzLaurer/synthetic_zeroshot_mixtral_v0.1, clinc_oos, go_emotions, tweet_eval sentiment and SetFit/20_newsgroups, all at pinned revisions and train files only. None is a BTZSC source dataset or a derivative of one. prepare_data.py asserts that no test or validation file is read, and uses a regex to drop synthetic rows whose prompts name excluded sources. The upstream checkpoint (deberta-v3-large-zeroshot-v2.0-c) is declared as trained on MNLI, FEVER-NLI and synthetic data. The trajectory downloads only these train parquets (trajectory#15, trajectory#20).
✓no public split tuningThe public validation split was not used for tuning. Head selection uses a selection partition carved from the training
The public validation split was not used for tuning. Head selection uses a selection partition carved from the training rows, and temperature calibration uses a calibration partition from the same rows (train.py, calibrate.py, prepare_data.py). /task/validate.py was run against the public validation file only once, at trajectory#82. That was after training and calibration finished and the classifier hashes were frozen (trajectory#79, evidence/frozen_classifier.json). Afterwards only the report's field names and runtime diagnostics were read (trajectory#83-85), and no weights or config changed before submission (trajectory#96-97). The earlier mention of the file is a `wc -c` byte count. evidence/public_validation_check.json reports diagnostics only.
✓no jev reproductionsNo JEV reproduction appears anywhere. The only models involved are MoritzLaurer/deberta-v3-large-zeroshot-v2.0-c at a pi
No JEV reproduction appears anywhere. The only models involved are MoritzLaurer/deberta-v3-large-zeroshot-v2.0-c at a pinned revision, which serves as the shipped backbone, and intfloat/e5-small-v2, which was tried as a retriever and abandoned (moved to /workspace/dev/unused at trajectory#46, not shipped). No Jeff, Laya, Open-Jev, jev-4b or typesafe-diffusion-lab artifacts appear in the code, the manifests or the trajectory.
✓stateless zero shot inferenceInference is stateless and zero-shot. server.py scores each request on its own: it tokenizes every (text, label) pair, t
Inference is stateless and zero-shot. server.py scores each request on its own: it tokenizes every (text, label) pair, takes FastDeberta features, multiplies them by the trained head weight, applies the adaptive temperature and a softmax, and returns probabilities in the order the labels arrived. The dynamic batching only groups concurrent requests. There is no cache, no cross-request state, no dataset lookup and no external call. fast_model.py precomputes only weight transforms that do not depend on the input. /health returns a static response.
✓actual inference uses reviewed weightsThe serving path loads the reviewed weights. serve.sh execs server.py. server.py loads the backbone from base_model/mode
The serving path loads the reviewed weights. serve.sh execs server.py. server.py loads the backbone from base_model/model.safetensors through FastDeberta, takes weight from head.safetensors and takes coefficients from calibration.safetensors. These match the inventoried tensors and the hashes in training_manifest.json. The trajectory shows the trained head and calibration files being copied into the server directory (trajectory#78), and the hashes were checked again before submission (trajectory#84).
✓harness credentials used only by the harnessThere is no sign of credential misuse. The shipped code and the trajectory contain no API_KEY, BASE_URL or OPENROUTER re
There is no sign of credential misuse. The shipped code and the trajectory contain no API_KEY, BASE_URL or OPENROUTER references used by the participant. External access in the session went through spin compute commands and Hugging Face Hub downloads, both of which the policy allows. The served code runs offline (HF_HUB_OFFLINE in serve.sh) and makes no network calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed40m 34s into the runtook 6m 22s
startup 75.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 75.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0001896784977072191, 0.00024236061916487714, 0.988472229183802, 0.0009600425548178292, 0.005966320118487272, 0.0002595619778452758, 0.0010495975252941263, 0.00021048855799188465, 0.00017644606467898257, 0.00019031094207616123, 0.0003 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 26.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.19, p95 0.23
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Official self-check passed: 164 documents/s, 75.5-second startup, and 4,400/4,400 valid responses.