Zero-shot classification
- Score
- 47.8
- Wallet spend
- $11.51 of $50
- Model cost
- $33.99
- Agent time
- 1h 17m
- Steps
- 210
- Tokens in / out
- 21.9M / 144k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 14
Score
47.8raw metric 0.6206
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.757 | 0.223 | 1.00 | 68.8 |
| amazonpolarity | 0.884 | 0.522 | 1.00 | 75.8 |
| appreviews | 0.857 | 0.521 | 1.00 | 70.2 |
| banking77 | 0.445 | 0.0142 | 1.00 | 43.7 |
| biasframes_intent | 0.443 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.585 | 0.456 | 1.00 | 23.7 |
| biasframes_sex | 0.635 | 0.369 | 1.00 | 42.1 |
| capsotu | 0.451 | 0.0166 | 1.00 | 44.2 |
| emotiondair | 0.486 | 0.132 | 1.00 | 40.8 |
| empathetic | 0.344 | 0.0300 | 1.00 | 32.4 |
| financialphrasebank | 0.438 | 0.300 | 1.00 | 19.7 |
| imdb | 0.899 | 0.510 | 1.00 | 79.4 |
| manifesto | 0.185 | 0.00957 | 1.00 | 17.8 |
| massive | 0.566 | 0.0199 | 1.00 | 55.7 |
| rottentomatoes | 0.807 | 0.509 | 1.00 | 60.7 |
| trueteacher | 0.237 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.808 | 0.511 | 1.00 | 60.7 |
| wikitoxic_obscene | 0.836 | 0.514 | 1.00 | 66.3 |
| wikitoxic_threat | 0.656 | 0.360 | 1.00 | 46.3 |
| wikitoxic_toxicaggregated | 0.822 | 0.508 | 1.00 | 63.8 |
| yahootopics | 0.593 | 0.0647 | 1.00 | 56.5 |
| yelpreviews | 0.917 | 0.513 | 1.00 | 83.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6804
- ece
- 0.0999
- accuracy
- 0.7293
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py runs real AdamW training (seed 44617, 8000 steps) on encoder layers 6 and above, the pooler and the classifier.
train.py runs real AdamW training (seed 44617, 8000 steps) on encoder layers 6 and above, the pooler and the classifier. The starting point is the pinned NLI checkpoint at revision b3546ea6b. Embeddings and layers 0-5 are frozen. evidence/weight_update.json shows that layer 0 and the embeddings are unchanged, while the changed tensors include layer 6 q_proj, layer 23 output, the pooler and the classifier. model/export.json traces the shipped FP16 model/model.safetensors (sha 78c1a637) back to checkpoint step-8000. The trajectory shows this run, and provenance_correction records that the final run restarted cleanly from the pinned checkpoint.
✓no evaluation data useprepare_data.py downloads only official train files, pinned to revisions, and asserts that 'train' appears in each filen
prepare_data.py downloads only official train files, pinned to revisions, and asserts that 'train' appears in each filename and that 'test' and 'validation' do not. The sources are go_emotions, tweet_sentiment_extraction, TweetEval offensive, OLID intersected with TweetEval train, clinc_oos, 20 newsgroups, enron_spam and MultiNLI. Labels come from handwritten templates. The TREC transfer check (prepare_transfer_check.py) uses the train_5500 file and is used for selection only. Every source is declared in the manifests and code, and none is a BTZSC evaluation source or the 500 public rows.
✓no public split tuningCheckpoint selection and temperature calibration in calibrate.py and evaluate_train_holdout.py use only a hash-based int
Checkpoint selection and temperature calibration in calibrate.py and evaluate_train_holdout.py use only a hash-based internal train holdout and the TREC train subset. validate.py ran once, at trajectory step 195, after the model and calibration were final. The step 197 write-up states that its accuracy was not used for any decision, and the configuration did not change afterwards.
✓no jev reproductionstraining_manifest.json lists teacher_models: [], and architecture.json declares no teachers and no JEV artifacts. The tr
training_manifest.json lists teacher_models: [], and architecture.json declares no teachers and no JEV artifacts. The training data consists of public train splits with template labels. No outputs from a reproduction of a judged or evaluated model were used.
✓stateless zero shot inferenceserver.py accepts only text and labels. inference.py scores each text-label pair independently with the cross-encoder en
server.py accepts only text and labels. inference.py scores each text-label pair independently with the cross-encoder entailment logit, then applies a fixed cardinality temperature taken from inference_config.json. Nothing is cached or adapted across requests. Batching only groups computation. The label-permutation test in the trajectory showed a maximum difference of 0.0.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and execs server.py. server.py builds the Classifier in inference.py, and
serve.sh installs the vendored wheels offline and execs server.py. server.py builds the Classifier in inference.py, and both inference.py and fast_model.py load AutoModelForSequenceClassification from model/ with local_files_only. That directory contains the reviewed model/model.safetensors: a standard DeBERTa-v3-large layout in F16 with a [3,1024] classifier and export sha 78c1a637. The trajectory at step 199 confirms that the serving sources match the checked server.
✓harness credentials used only by the harnessThe participant listed environment variable names only, with no values, at step 7. Searches found no OpenAI or OpenRoute
The participant listed environment variable names only, with no values, at step 7. Searches found no OpenAI or OpenRouter API calls and no use of a base URL. The only credential use allowed was HF_TOKEN for Hub downloads, which the task permits. The served model runs offline.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed22m 6s into the runtook 4m 47s
startup 56.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 56.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0008496665262269235, 0.000940495110991676, 0.8719066988369557, 0.002473039711501905, 0.012583685475430857, 0.0014595151571697752, 0.0008397676113608264, 0.001014933452634867, 0.0005851083685261754, 0.0007935246270144914, 0.0135135141 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 27.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 0.22
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed1h 14m into the runtook 3m 13s
startup 66.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 66.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00017456095048630808, 0.00026199734420073023, 0.9810712503865424, 0.0007002433846544798, 0.0035156642662089923, 0.0003108489699502969, 0.00016372024811664839, 0.00031756380762938105, 0.00011630471113225308, 0.00018218400031407383, 0. ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 27.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 0.25
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Final offline H100 selfcheck passed:
- Startup: 66.7 seconds
- Throughput: 159.2 documents/second
- Valid responses: 4,400/4,400
Training code, weights, calibration, provenance, and offline dependencies are included.