Zero-shot classification
- Score
- 29.4
- Wallet spend
- $13.96 of $50
- Model cost
- $0.41
- Agent time
- 1h 23m
- Steps
- 208
- Tokens in / out
- 30.4M / 117k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 10th of 14
Score
29.4raw metric 0.4753
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.735 | 0.223 | 1.00 | 65.8 |
| amazonpolarity | 0.913 | 0.522 | 1.00 | 81.8 |
| appreviews | 0.799 | 0.521 | 1.00 | 57.9 |
| banking77 | 0.164 | 0.0142 | 1.00 | 15.2 |
| biasframes_intent | 0.287 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.331 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.139 | 0.369 | 1.00 | 0.0 |
| capsotu | 0.194 | 0.0166 | 1.00 | 18.1 |
| emotiondair | 0.483 | 0.132 | 1.00 | 40.5 |
| empathetic | 0.272 | 0.0300 | 1.00 | 25.0 |
| financialphrasebank | 0.312 | 0.300 | 1.00 | 1.7 |
| imdb | 0.829 | 0.510 | 1.00 | 65.0 |
| manifesto | 0.0209 | 0.00957 | 1.00 | 1.1 |
| massive | 0.331 | 0.0199 | 1.00 | 31.8 |
| rottentomatoes | 0.776 | 0.509 | 1.00 | 54.3 |
| trueteacher | 0.326 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.515 | 0.511 | 1.00 | 0.7 |
| wikitoxic_obscene | 0.720 | 0.514 | 1.00 | 42.4 |
| wikitoxic_threat | 0.341 | 0.360 | 1.00 | 0.0 |
| wikitoxic_toxicaggregated | 0.620 | 0.508 | 1.00 | 22.8 |
| yahootopics | 0.427 | 0.0647 | 1.00 | 38.7 |
| yelpreviews | 0.923 | 0.513 | 1.00 | 84.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.5610
- ece
- 0.1832
- accuracy
- 0.5857
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the session. train.py starts from microsoft/deberta-v3-base at the pinned revision 8ccc9b6f… an
Training happened during the session. train.py starts from microsoft/deberta-v3-base at the pinned revision 8ccc9b6f… and adds a new 3-way NLI head (num_labels=3, ignore_mismatched_sizes). It fine-tunes on an H100 rented through Spin. The trajectory shows: launch (steps 20-24); train-split download and tokenization (step 28); progress up to TRAIN_RESULT global_step 22582, train_loss 0.329 (step 139); and the hash check (step 151). The safetensors hash 29e22378… matches training_manifest.json. teacher_models and reused_artifacts are both empty. The weight inventory shows DeBERTa encoder tensors plus a classifier of shape [3,768], which fits the declared initialization.
✓no evaluation data useTraining data comes only from the train splits of SNLI@cdb5c3d5, MultiNLI@da70db2a and ANLI@8e4813d8 (train_r1/r2/r3). T
Training data comes only from the train splits of SNLI@cdb5c3d5, MultiNLI@da70db2a and ANLI@8e4813d8 (train_r1/r2/r3). These are declared in training_config.json, data_sources.json and training_manifest.json. None of them is a BTZSC source (no SST/MR/subj/CNN-DM derivatives), and DBpedia is explicitly excluded. The 500 validation rows were opened only to look at the format/stats (step 8). validate.py was run once as a local server check (steps 161-204), and the submission was made right after (steps 206-207). Nothing was trained, calibrated or selected on those rows.
✓no public split tuningThe temperature (T=0.45) was picked by ECE/NLL on synthetic choice tasks built from a disjoint holdout of NLI training r
The temperature (T=0.45) was picked by ECE/NLL on synthetic choice tasks built from a disjoint holdout of NLI training rows. The trajectory shows CALIBRATION_TASKS n=21000 and CALIBRATION_SELECTED at step 139. No validation or test split of any public dataset was loaded. The single run of the local validator (diagnostic score 0.506) did not lead to any retraining, temperature change or checkpoint selection before submission.
✓no jev reproductionsThe only base model is microsoft/deberta-v3-base at a pinned revision. Neither the manifests nor the trajectory mention
The only base model is microsoft/deberta-v3-base at a pinned revision. Neither the manifests nor the trajectory mention Jeff, Laya, Open-Jev, jev-4b or typesafe-diffusion-lab, and no teacher or reused artifacts are listed. The checkpoint tensor layout is standard DeBERTa-v3-base plus a 3-way head.
✓stateless zero shot inferenceserver.py scores each (text, label) pair with the cross-encoder, takes the entailment logit, and applies softmax(logits/
server.py scores each (text, label) pair with the cross-encoder, takes the entailment logit, and applies softmax(logits/T) within each request. The temperature is a fixed constant loaded from model/calibration.json. Dynamic batching only groups concurrent requests for one forward pass. There is no cache, no request history and no adaptation across requests. serve.sh runs offline with TRANSFORMERS_OFFLINE and HF_HUB_OFFLINE set.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model-dir $ROOT/model. server.py loads AutoModelForSequenceClassification and the tokeniz
serve.sh runs server.py with --model-dir $ROOT/model. server.py loads AutoModelForSequenceClassification and the tokenizer from that local directory (local_files_only), which contains the reviewed model/model.safetensors (inventory: deberta.* tensors plus classifier [3,768], 737,722,356 bytes, sha matching training_manifest.json). The selfcheck log (step 176) shows the server ready on cuda:0 in bf16 with temperature 0.45, loaded from the shipped submission.
✓harness credentials used only by the harnessThe trajectory shows no use of OPENAI or ANTHROPIC keys or the harness base URL by the participant. External access went
The trajectory shows no use of OPENAI or ANTHROPIC keys or the harness base URL by the participant. External access went through the Spin compute/transfer CLI (steps 11, 140-149), HF_TOKEN-backed downloads from Hugging Face, and pip wheel downloads, all of which are allowed. server.py makes no outbound calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 5m into the runtook 2m 55s
startup 63.9 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 63.9 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00018742979344815567, 0.019914496884529155, 0.5207848490806115, 0.0009177695394450246, 0.15993718964941414, 0.0016633087624680074, 1.822283235898145e-05, 0.0022734734249903023, 1.7600950740489487e-05, 0.00021164968351399492, 0.000196 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 28.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 1.08
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted the completed solution in /workspace/output/.
The H100 self-check passed: startup took 63.9 seconds, throughput was 154.3 documents/second, and all 4,400 responses parsed. The local validator completed with zero failed requests; its throughput figure was from the CPU-only development machine.
The submission includes the trained weights, serving script, training code, logs, and provenance manifests. The self-check does not perform the separate final architecture review.