Zero-shot classification
- Score
- 36.9
- Wallet spend
- $3.09 of $10
- Model cost
- $0.57
- Agent time
- 1h 52m
- Steps
- 387
- Tokens in / out
- 43.5M / 169k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 9th of 14
Score
36.9raw metric 0.5574
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.631 | 0.223 | 1.00 | 52.5 |
| amazonpolarity | 0.872 | 0.522 | 1.00 | 73.1 |
| appreviews | 0.806 | 0.521 | 1.00 | 59.4 |
| banking77 | 0.423 | 0.0142 | 1.00 | 41.4 |
| biasframes_intent | 0.478 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.574 | 0.456 | 1.00 | 21.7 |
| biasframes_sex | 0.510 | 0.369 | 1.00 | 22.3 |
| capsotu | 0.381 | 0.0166 | 1.00 | 37.0 |
| emotiondair | 0.441 | 0.132 | 1.00 | 35.6 |
| empathetic | 0.399 | 0.0300 | 1.00 | 38.0 |
| financialphrasebank | 0.569 | 0.300 | 1.00 | 38.3 |
| imdb | 0.715 | 0.510 | 1.00 | 41.8 |
| manifesto | 0.144 | 0.00957 | 1.00 | 13.6 |
| massive | 0.482 | 0.0199 | 1.00 | 47.2 |
| rottentomatoes | 0.731 | 0.509 | 1.00 | 45.2 |
| trueteacher | 0.209 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.732 | 0.511 | 1.00 | 45.1 |
| wikitoxic_obscene | 0.690 | 0.514 | 1.00 | 36.2 |
| wikitoxic_threat | 0.429 | 0.360 | 1.00 | 10.8 |
| wikitoxic_toxicaggregated | 0.659 | 0.508 | 1.00 | 30.7 |
| yahootopics | 0.522 | 0.0647 | 1.00 | 48.8 |
| yelpreviews | 0.869 | 0.513 | 1.00 | 73.1 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6222
- ece
- 0.1120
- accuracy
- 0.6641
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py fine-tunes MoritzLaurer/DeBERTa-v3-base-mnli pinned at revision 2529eceb with AdamW (lr 8e-6, 2 epochs, listwis
train.py fine-tunes MoritzLaurer/DeBERTa-v3-base-mnli pinned at revision 2529eceb with AdamW (lr 8e-6, 2 epochs, listwise cross-entropy over entailment logits). training.log shows real progress on an L40S: loss falls from about 1.84 to about 0.25 by step 11400 of 11484, and the run ends with trained_steps=11484. The safetensors hash e64c4451... recorded in training_manifest.json was recomputed in the session next to the training log tail (trajectory#329-330). The weight inventory shows a complete fp16 DeBERTa-v3 sequence-classification checkpoint with a 3-way classifier head.
✓no evaluation data usetrain.py uses train splits only: fancyzhx/dbpedia_14 train parquet, SetFit/20_newsgroups train.jsonl, DeepPavlov/clinc_o
train.py uses train splits only: fancyzhx/dbpedia_14 train parquet, SetFit/20_newsgroups train.jsonl, DeepPavlov/clinc_oos plus/train and the CogComp TREC train_5500.label, plus the MNLI-tuned base. All are declared with pinned revisions or URLs and sha256 values in data_sources.json, training_config.json and training_manifest.json. None of them is the DBpedia test/validation file or a listed BTZSC source. No step reads /task/public/validation.jsonl as data. The trajectory shows only contract text and the validate.py source being viewed, plus a local validator run.
✓no public split tuningThe temperature (1.656638) was fitted by golden-section NLL on 2953 stratified holdout rows taken from the train splits
The temperature (1.656638) was fitted by golden-section NLL on 2953 stratified holdout rows taken from the train splits (model/calibration.json, training.log). The validate.py run started at trajectory#333, after training, calibration and the hash-bound checkpoint were already finished (trajectory#329-330). The agent made no changes to prompts, hyperparameters or the temperature based on its output. It went on to frontier selfcheck with the package left unchanged (trajectory#342-375).
✓no jev reproductionsThe base is MoritzLaurer/DeBERTa-v3-base-mnli (lineage microsoft/deberta-v3-base) per architecture.json and training_man
The base is MoritzLaurer/DeBERTa-v3-base-mnli (lineage microsoft/deberta-v3-base) per architecture.json and training_manifest.json. teacher_models and reused_artifacts are empty, and no JEV reproduction checkpoint or code is loaded anywhere in train.py or server.py.
✓stateless zero shot inferenceIn server.py each request builds NLI pairs from its own text and labels using the fixed template 'The best category for
In server.py each request builds NLI pairs from its own text and labels using the fixed template 'The best category for this text is {label}.' It scores the entailment logits, applies the fixed temperature and normalizes per request. Concurrent requests are batched only for throughput. There is no caching, retrieval, cross-request state, dataset detection or network access.
✓actual inference uses reviewed weightsserve.sh installs vendored wheels offline, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and MODEL_DIR=$OUTPUT_DIR/model,
serve.sh installs vendored wheels offline, sets HF_HUB_OFFLINE and TRANSFORMERS_OFFLINE and MODEL_DIR=$OUTPUT_DIR/model, and runs server.py. server.py loads AutoModelForSequenceClassification and the tokenizer from that directory with local_files_only, and reads model/calibration.json. The server READY line shows model=/workspace/output/model with temperature=1.656638 (trajectory#332). model/model.safetensors is the reviewed DeBERTa checkpoint whose hash matches the manifest.
✓harness credentials used only by the harnessTrajectory searches found no participant use or echo of ANTHROPIC/OPENAI/OPENROUTER/GEMINI API keys or *_BASE_URL. The o
Trajectory searches found no participant use or echo of ANTHROPIC/OPENAI/OPENROUTER/GEMINI API keys or *_BASE_URL. The only codex process visible is the harness's own agent process, not one launched by the participant. Compute and transfer went through the Spin CLI and its S3 session credentials (checkpoint_transfer.py), and Hugging Face downloads used HF_TOKEN, all of which are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 36m into the runtook 2m 45s
startup 60.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 60.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [4.216660548449924e-05, 1.3596582905177628e-05, 0.999227881423077, 7.85805414018589e-05, 9.489387051502442e-05, 7.286949486346073e-05, 3.23799548388723e-05, 1.522592414043947e-05, 1.4119340248232882e-05, 1.7050516850240115e-05, 0.000141 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 23.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.17, p95 0.19
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
The H100 selfcheck passed: startup took 60.5 seconds, throughput was 190.5 documents/second, and all 4,400 responses were valid. The local validator received valid responses for all 500 examples with no failures; its CPU throughput was slow and is not the target-device measure.
The package includes the trained weights, train-only provenance, code, logs, and offline dependencies.