Zero-shot classification
- Score
- 39.8
- Wallet spend
- $14.41 of $50
- Model cost
- $0.66
- Agent time
- 4h 10m
- Steps
- 286
- Tokens in / out
- 27.0M / 248k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 11th of 19
Score
39.8raw metric 0.5716
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.787 | 0.223 | 1.00 | 72.5 |
| amazonpolarity | 0.833 | 0.522 | 1.00 | 65.0 |
| appreviews | 0.838 | 0.521 | 1.00 | 66.2 |
| banking77 | 0.489 | 0.0142 | 1.00 | 48.2 |
| biasframes_intent | 0.439 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.492 | 0.456 | 1.00 | 6.6 |
| biasframes_sex | 0.638 | 0.369 | 1.00 | 42.6 |
| capsotu | 0.321 | 0.0166 | 1.00 | 30.9 |
| emotiondair | 0.429 | 0.132 | 1.00 | 34.2 |
| empathetic | 0.363 | 0.0300 | 1.00 | 34.3 |
| financialphrasebank | 0.366 | 0.300 | 1.00 | 9.3 |
| imdb | 0.792 | 0.510 | 1.00 | 57.7 |
| manifesto | 0.0536 | 0.00957 | 1.00 | 4.4 |
| massive | 0.385 | 0.0199 | 1.00 | 37.2 |
| rottentomatoes | 0.769 | 0.509 | 1.00 | 53.0 |
| trueteacher | 0.301 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.791 | 0.511 | 1.00 | 57.2 |
| wikitoxic_obscene | 0.801 | 0.514 | 1.00 | 59.1 |
| wikitoxic_threat | 0.550 | 0.360 | 1.00 | 29.7 |
| wikitoxic_toxicaggregated | 0.737 | 0.508 | 1.00 | 46.6 |
| yahootopics | 0.550 | 0.0647 | 1.00 | 51.9 |
| yelpreviews | 0.850 | 0.513 | 1.00 | 69.2 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6432
- ece
- 0.1152
- accuracy
- 0.6886
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the task. A full fine-tune (train.py, AdamW at 2e-5, mixed NLI and K-way CE batches) of the dec
Training happened during the task. A full fine-tune (train.py, AdamW at 2e-5, mixed NLI and K-way CE batches) of the declared base MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli was run for 48000 steps. The run was launched at trajectory#250 and finished at trajectory#270, with the end of training.log showing step 48000 and 'done'. The step-32000 checkpoint was copied into output/model at trajectory#275. training_config.json and training_manifest.json agree with this run (48000 steps, save_every 8000, selected 32000).
✓no evaluation data useNo evaluation data was used. data_manifest.json and data_build/sources.py list only train splits: dbpedia_14 train plus
No evaluation data was used. data_manifest.json and data_build/sources.py list only train splits: dbpedia_14 train plus other public classification corpora, and NLI train sets (multi_nli, snli, anli, fever, lingnli). data_build/prep_v3.py builds cls_train and the proxy sets from held-out train rows. /task/public/validation.jsonl (DBpedia-14 test rows) was used only for a local validate.py server check at trajectory#252, and not for training, calibration or selection. All sources are declared public; exact download logs were not all retained, so they are unverified.
✓no public split tuningNo public validation or test split was used for tuning. Temperature calibration (calibrate.py, calibration.json: a=1.2,
No public validation or test split was used for tuning. Temperature calibration (calibrate.py, calibration.json: a=1.2, b=-0.28) was fitted on proxy_calib (trajectory#272-#274). Checkpoint selection used proxy_eval/proxy_eval_v2, both built from held-out train-split rows (trajectory#260, #271). None of these used a public validation or test split.
✓no jev reproductionsThere is no evidence of JEV reproductions. training_manifest.json lists no teacher models and no reused artifacts. The b
There is no evidence of JEV reproductions. training_manifest.json lists no teacher models and no reused artifacts. The base is a declared public NLI checkpoint at a pinned revision (architecture.json). The earlier ModernBERT/zeroshot-v2.0 approach was stopped as non-compliant at trajectory#123 and was not shipped.
✓stateless zero shot inferenceInference is stateless and zero-shot. server.py scores each request independently: the NLI cross-encoder gives an entail
Inference is stateless and zero-shot. server.py scores each request independently: the NLI cross-encoder gives an entailment logit per text/label-description pair, followed by a temperature softmax with T(K). It only does dynamic batching, with no caching, retrieval, dataset access or adaptation across requests. serve.sh runs offline (HF_HUB_OFFLINE=1).
✓actual inference uses reviewed weightsThe server uses the reviewed weights. serve.sh sets MODEL_DIR=$HERE/model and execs server.py, which loads AutoModelForS
The server uses the reviewed weights. serve.sh sets MODEL_DIR=$HERE/model and execs server.py, which loads AutoModelForSequenceClassification from MODEL_DIR with local_files_only. model/model.safetensors (inventory: DeBERTa-v2 tensors plus a 3-way classifier) matches model/config.json. Its sha256 0448b3fd… was recorded in the manifest regenerated at trajectory#277 after the step-32000 copy at trajectory#275.
✓harness credentials used only by the harnessCredentials were not misused. The trajectory shows HF_TOKEN used only for Hugging Face dataset and model downloads durin
Credentials were not misused. The trajectory shows HF_TOKEN used only for Hugging Face dataset and model downloads during data preparation and the base-model snapshot. There were no calls to model APIs and no misuse of harness credentials, and the shipped serving code uses no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 18m into the runtook 2m 54s
startup 68.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 68.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [8.372221562442127e-06, 8.035180792785518e-06, 0.999448239413197, 6.0173267791436746e-05, 1.0712999225103864e-05, 7.699702718254301e-05, 6.2795032910895694e-06, 8.035180792785518e-06, 6.2795032910895694e-06, 6.2795032910895694e-06, 0.00 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 23.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.17, p95 1.10
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed4h 7m into the runtook 2m 59s
startup 69.8 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 69.8 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.1978970966567609e-05, 7.764846610921103e-06, 0.998958340157512, 0.00012255203720401155, 1.776598033995249e-05, 0.00013793403296739803, 1.641932291842987e-05, 1.1070967589008142e-05, 5.44603192145508e-06, 6.129585143859999e-06, 0.0003 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 25.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.18, p95 1.12
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission — /workspace/output/ (76 files, 4.75 GB)
| Piece | Choice |
|---|---|
| Base model | MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli @ 6f5cf0a2… (MNLI+FEVER+ANLI only; no excluded data) |
| Method | Cross-encoder: premise = text, hypothesis = label description; entailment logit → single-label distribution |
| Training | 48,000 steps, K-way CE over gold + K−1 same-dataset distractors mixed 60/40 with 3-way NLI CE, on 694k rendered label tasks + 1.15M NLI pairs (training splits only) |
| Exported checkpoint | step 32,000, selected on two held-out proxy task sets |
| Calibration | T(K)=exp(1.2 − 0.28·lnK) on the entailment logit, fitted to maximise macro-F1·(1−ECE) on a held-out proxy calibration set |
| Serving | serve.sh installs vendored wheels offline into /tmp/pylibs, bf16 on one H100, dynamic batching across requests |
Verified with frontier selfcheck on a clean H100: startup 69.8 s (limit 300), throughput 170.7 docs/s (limit 50), all 4,400 responses valid, healthy after load. Proxy scores: 0.7855 / 0.7860 macro-F1·(1−ECE) on the two held-out evaluation sets.
Compliance notes: no BTZSC source dataset or its val/test split was used; the 500 public examples were only used with validate.py/selfcheck checks; manifests pin every corpus by 40-char commit or sha256: content digest, and no corpora are packaged — only provenance, build code and logs.