Zero-shot classification
- Score
- 38.7
- Wallet spend
- $5.29 of $10
- Model cost
- $4.92
- Agent time
- 1h 52m
- Steps
- 184
- Tokens in / out
- 17.1M / 221k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 8th of 14
Score
38.7raw metric 0.5536
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.679 | 0.223 | 1.00 | 58.7 |
| amazonpolarity | 0.861 | 0.522 | 1.00 | 70.9 |
| appreviews | 0.834 | 0.521 | 1.00 | 65.3 |
| banking77 | 0.487 | 0.0142 | 1.00 | 47.9 |
| biasframes_intent | 0.297 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.409 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.557 | 0.369 | 1.00 | 29.8 |
| capsotu | 0.384 | 0.0166 | 1.00 | 37.4 |
| emotiondair | 0.460 | 0.132 | 1.00 | 37.7 |
| empathetic | 0.302 | 0.0300 | 1.00 | 28.0 |
| financialphrasebank | 0.416 | 0.300 | 1.00 | 16.5 |
| imdb | 0.836 | 0.510 | 1.00 | 66.5 |
| manifesto | 0.108 | 0.00957 | 1.00 | 9.9 |
| massive | 0.418 | 0.0199 | 1.00 | 40.6 |
| rottentomatoes | 0.727 | 0.509 | 1.00 | 44.3 |
| trueteacher | 0.277 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.778 | 0.511 | 1.00 | 54.6 |
| wikitoxic_obscene | 0.829 | 0.514 | 1.00 | 64.9 |
| wikitoxic_threat | 0.467 | 0.360 | 1.00 | 16.8 |
| wikitoxic_toxicaggregated | 0.720 | 0.508 | 1.00 | 43.1 |
| yahootopics | 0.427 | 0.0647 | 1.00 | 38.7 |
| yelpreviews | 0.905 | 0.513 | 1.00 | 80.5 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6216
- ece
- 0.1246
- accuracy
- 0.6720
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the task. train.py downloads cross-encoder/nli-deberta-v3-large at the pinned revision bab4bc71
Training happened during the task. train.py downloads cross-encoder/nli-deberta-v3-large at the pinned revision bab4bc71…, freezes only the embeddings and fine-tunes the rest with AdamW and a cosine schedule. training_large.log records this initialization (base weight sha 01cb91b0…). It then shows 6000 training steps, a checkpoint saved at step 6000, and a final model sha256 of be4cd9a2…. That hash matches training_manifest.json and the shipped model/model.safetensors, whose inventory holds full DeBERTa-v3-large tensors plus a 3x1024 classifier. trajectory#172 confirms the shipped weights match the remotely trained model.
✓no evaluation data useAll data comes from 8 declared, revision-pinned, sha256-hashed train files on the Hugging Face Hub: clinc_oos plus, go_e
All data comes from 8 declared, revision-pinned, sha256-hashed train files on the Hugging Face Hub: clinc_oos plus, go_emotions simplified, SetFit 20_newsgroups, four tweet_eval subsets and MultiNLI. data.py raises an error on any file name that lacks 'train' or contains 'test' or 'validation'. architecture.json and training_manifest.json declare no teachers. The downloads and hashes are logged in training_large.log and evidence/data_manifest.json. I found no use of BTZSC evaluation data.
✓no public split tuningCheckpoint selection and temperature calibration use the selection and calibration partitions that data.py carves from t
Checkpoint selection and temperature calibration use the selection and calibration partitions that data.py carves from train files (train.py, calibrate_counts.py). The public validation file was used once, at trajectory#174–177, after the prediction freeze recorded in evidence/prediction_freeze.json. Only non-score server-check fields were printed. trajectory#179 verifies the predictive files were unchanged afterwards.
✓no jev reproductionsThe only reused code is PyPI transformers and torch, as declared in architecture.json and training_manifest.json. The ba
The only reused code is PyPI transformers and torch, as declared in architecture.json and training_manifest.json. The base is a public NLI cross-encoder and no teachers are used. Trajectory searches found no references to JEV artifacts, jeff, tasksource or dbpedia. engine.py and train.py are original implementations.
✓stateless zero shot inferenceserver.py batches independent requests through a worker queue. engine.py tokenizes each text/label pair with a fixed hyp
server.py batches independent requests through a worker queue. engine.py tokenizes each text/label pair with a fixed hypothesis template. The score is the entailment logit minus the contradiction logit, then softmax with a temperature that depends only on the label count (inference_config.json). There is no cache, no cross-request state, no adaptation and no external calls. Static positions and CUDA graphs depend only on shapes (formatting.py, engine.py).
✓actual inference uses reviewed weightsserve.sh runs `exec python server.py` with offline environment variables. server.py initializes Classifier(ROOT), and en
serve.sh runs `exec python server.py` with offline environment variables. server.py initializes Classifier(ROOT), and engine.py loads root/model with local_files_only. That directory holds the reviewed model/model.safetensors, whose sha256 be4cd9a2… matches training_manifest.json and trajectory#172. The wheels are trusted, upstream-approved builds.
✓harness credentials used only by the harnessTrajectory searches found no participant use of harness API keys or base URLs. The API_KEY and BASE_URL hits were docume
Trajectory searches found no participant use of harness API keys or base URLs. The API_KEY and BASE_URL hits were documentation text only, and the openrouter hit was help text. No codex or claude worker processes were launched. The shipped code makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 35m into the runtook 3m 18s
startup 71.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 71.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0019687922561702903, 0.0009942088052523961, 0.5475827195317977, 0.005124023286341481, 0.006968448969922176, 0.0234324761704317, 0.0015238037385319034, 0.002866798380916635, 0.0005284567985002613, 0.0012203777792390888, 0.024454755608 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 37.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.27, p95 0.31
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm.
- Trained classifier, weights, provenance, logs, and offline dependencies included.
- Training, calibration, and selection use only permitted original training data.
- Local 500-example check: zero response failures.
- Offline H100 self-check passed: 71-second startup, 116.5 documents/second, and all 4,400 responses valid.
Entry point: /workspace/output/serve.sh:1.