Zero-shot classification
- Score
- 43.2
- Wallet spend
- $19.43 of $50
- Model cost
- $8.88
- Agent time
- 4h 56m
- Steps
- 272
- Tokens in / out
- 26.0M / 284k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 6th of 14
Score
43.2raw metric 0.5891
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.681 | 0.223 | 1.00 | 58.9 |
| amazonpolarity | 0.857 | 0.522 | 1.00 | 70.1 |
| appreviews | 0.842 | 0.521 | 1.00 | 67.0 |
| banking77 | 0.376 | 0.0142 | 1.00 | 36.7 |
| biasframes_intent | 0.749 | 0.527 | 1.00 | 46.8 |
| biasframes_offensive | 0.623 | 0.456 | 1.00 | 30.8 |
| biasframes_sex | 0.607 | 0.369 | 1.00 | 37.7 |
| capsotu | 0.385 | 0.0166 | 1.00 | 37.5 |
| emotiondair | 0.440 | 0.132 | 1.00 | 35.5 |
| empathetic | 0.281 | 0.0300 | 1.00 | 25.9 |
| financialphrasebank | 0.692 | 0.300 | 1.00 | 55.9 |
| imdb | 0.854 | 0.510 | 1.00 | 70.1 |
| manifesto | 0.110 | 0.00957 | 1.00 | 10.2 |
| massive | 0.377 | 0.0199 | 1.00 | 36.5 |
| rottentomatoes | 0.726 | 0.509 | 1.00 | 44.2 |
| trueteacher | 0.236 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.765 | 0.511 | 1.00 | 52.0 |
| wikitoxic_obscene | 0.725 | 0.514 | 1.00 | 43.5 |
| wikitoxic_threat | 0.489 | 0.360 | 1.00 | 20.2 |
| wikitoxic_toxicaggregated | 0.726 | 0.508 | 1.00 | 44.4 |
| yahootopics | 0.531 | 0.0647 | 1.00 | 49.9 |
| yelpreviews | 0.887 | 0.513 | 1.00 | 76.7 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6516
- ece
- 0.1021
- accuracy
- 0.6995
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtraining/train.py is a real DDP fine-tuning loop. It starts from the pinned MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli
training/train.py is a real DDP fine-tuning loop. It starts from the pinned MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli revision 6f5cf0a2..., and the download is logged in training/model_download.log and trajectory#213. The trajectory shows the torchrun training runs. export_model.py writes out the selected checkpoint. The sha256 of the resulting model.safetensors, computed at trajectory#248, matches training_manifest.json (cf6cd57d...). The manifest reports 2976 training steps, and checkpoint selection chose step 1750.
✓no evaluation data usetraining/data_log.json and prep_data.py load only train splits, each at a pinned revision: MNLI, SNLI, ANLI, nli_fever,
training/data_log.json and prep_data.py load only train splits, each at a pinned revision: MNLI, SNLI, ANLI, nli_fever, WANLI, PAWS, dbpedia_14, clinc_oos, atis, TREC, go_emotions, tweet_eval, toxigen, fiqa and xsum, plus synthetic pairs generated by prep_data.py. Every source is declared, and none is an evaluation-set source or derivative. The manifest lists no teacher models and no reused artifacts.
✓no public split tuningCheckpoint selection (selection_summary.json) and temperature calibration (calibrate.py) use a dev set that prep_data.py
Checkpoint selection (selection_summary.json) and temperature calibration (calibrate.py) use a dev set that prep_data.py carves out of the train splits. validate.py ran on validation.jsonl only at trajectory#250-251, after selection and calibration were finished, and nothing was tuned on its results.
✓no jev reproductionsNothing reproduces jev in the evidence: no matching references or artifacts were found. training_manifest.json declares
Nothing reproduces jev in the evidence: no matching references or artifacts were found. training_manifest.json declares no reused artifacts or teachers. The model is a fresh fine-tune of a declared public NLI base.
✓stateless zero shot inferenceserver.py scores each request on its own: it computes the entail-minus-contra logit difference and applies a fixed tempe
server.py scores each request on its own: it computes the entail-minus-contra logit difference and applies a fixed temperature softmax per label-count bucket. Batching only groups independent requests. There is no cache, retrieval, cross-request memory, stored examples or external calls.
✓actual inference uses reviewed weightsserve.sh sets HF_HUB_OFFLINE=1 and runs server.py with --model-dir model. server.py loads AutoModelForSequenceClassifica
serve.sh sets HF_HUB_OFFLINE=1 and runs server.py with --model-dir model. server.py loads AutoModelForSequenceClassification and serving_config.json from that directory, so it serves model/model.safetensors. That file's hash matches the trained export (trajectory#248, trajectory#253).
✓harness credentials used only by the harnessSearches of the trajectory and code for API keys, base URLs and LLM provider endpoints found only harness documentation.
Searches of the trajectory and code for API keys, base URLs and LLM provider endpoints found only harness documentation. The participant made no LLM API calls, and the server makes no external calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed4h 41m into the runtook 4m 38s
startup 71.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 71.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0002217602723043893, 0.00024125377672253894, 0.9960207946514077, 0.000526946694729699, 0.0008719213913891909, 0.00022865954604466929, 0.0002251834878401698, 0.00024310870977470092, 0.00023218926262686315, 0.0002375862623569719, 0.000 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 19.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.14, p95 0.16
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed4h 48m into the runtook 3m 7s
startup 72.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 72.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0002217602723043893, 0.00024125377672253894, 0.9960207946514077, 0.000526946694729699, 0.0008719213913891909, 0.00022865954604466929, 0.0002251834878401698, 0.00024310870977470092, 0.00023218926262686315, 0.0002375862623569719, 0.000 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 19.7 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.14, p95 0.17
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed4h 52m into the runtook 2m 46s
startup 78.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 78.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0002217602723043893, 0.00024125377672253894, 0.9960207946514077, 0.000526946694729699, 0.0008719213913891909, 0.00022865954604466929, 0.0002251834878401698, 0.00024310870977470092, 0.00023218926262686315, 0.0002375862623569719, 0.000 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 20.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.15, p95 0.18
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
105 files exactly matching selfcheck 3's digest. Submit: