Zero-shot classification
- Score
- 40.2
- Wallet spend
- $28.31 of $50
- Model cost
- $17.62
- Agent time
- 6h 15m
- Steps
- 267
- Tokens in / out
- 47.7M / 168k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 8th of 14
Score
40.2raw metric 0.5768
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.839 | 0.223 | 1.00 | 79.2 |
| amazonpolarity | 0.817 | 0.522 | 1.00 | 61.7 |
| appreviews | 0.847 | 0.521 | 1.00 | 68.0 |
| banking77 | 0.280 | 0.0142 | 1.00 | 27.0 |
| biasframes_intent | 0.511 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.546 | 0.456 | 1.00 | 16.6 |
| biasframes_sex | 0.533 | 0.369 | 1.00 | 25.9 |
| capsotu | 0.529 | 0.0166 | 1.00 | 52.1 |
| emotiondair | 0.452 | 0.132 | 1.00 | 36.8 |
| empathetic | 0.213 | 0.0300 | 1.00 | 18.9 |
| financialphrasebank | 0.643 | 0.300 | 1.00 | 48.9 |
| imdb | 0.814 | 0.510 | 1.00 | 62.2 |
| manifesto | 0.155 | 0.00957 | 1.00 | 14.7 |
| massive | 0.404 | 0.0199 | 1.00 | 39.2 |
| rottentomatoes | 0.807 | 0.509 | 1.00 | 60.6 |
| trueteacher | 0.232 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.777 | 0.511 | 1.00 | 54.4 |
| wikitoxic_obscene | 0.787 | 0.514 | 1.00 | 56.3 |
| wikitoxic_threat | 0.612 | 0.360 | 1.00 | 39.5 |
| wikitoxic_toxicaggregated | 0.762 | 0.508 | 1.00 | 51.6 |
| yahootopics | 0.494 | 0.0647 | 1.00 | 45.9 |
| yelpreviews | 0.634 | 0.513 | 1.00 | 24.8 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6644
- ece
- 0.1464
- accuracy
- 0.7152
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtraining_src/train.py starts from AutoModel.from_pretrained of the declared general encoder (answerdotai/ModernBERT-larg
training_src/train.py starts from AutoModel.from_pretrained of the declared general encoder (answerdotai/ModernBERT-large @45bb4654) and adds a newly initialised BiEncoderClassifier head (text_proj, label_proj, logit_scale), then trains it with listwise cross-entropy. training.log and training_model2.log show genuine runs: the loss falls, EVAL lines appear, checkpoints are saved, and the step counts (about 15k) match training_manifest.json. The trajectory shows the base model downloaded and train.py launched during the session (v5l run). The trajectory also shows the earlier v4l/v3mb runs being packaged. Both weight inventories contain the backbone tensors plus the task-specific projection head and scalar logit_scale. training_manifest.json lists reused_artifacts as empty.
✓no evaluation data usetraining_src/prep_data.py and DATA_SOURCES.md use only train splits of allowed corpora. They explicitly exclude the BTZS
training_src/prep_data.py and DATA_SOURCES.md use only train splits of allowed corpora. They explicitly exclude the BTZSC source datasets and their derivatives. The synthetic corpora (gen_synth.py, gen_synth2.py) are written by deepseek-chat-v3-0324 through the Spin proxy from author-chosen generic topic and label descriptions, not from evaluation data. I found no BTZSC test data in the code, the manifests or the trajectory. Dataset revisions are declared in training_manifest.json and training_src/dataset_revisions.json.
✓no public split tuningModel selection and calibration used dev sets carved from train splits (dev.jsonl, built by prep_data.py; eval_dev.py; e
Model selection and calibration used dev sets carved from train splits (dev.jsonl, built by prep_data.py; eval_dev.py; eval_ens.py on /root/data4), per training_config.json and selection_summary.json. The public validation.jsonl was uploaded and passed only to /task/validate.py, as an interface and throughput smoke test. That run exposed a serving bug (uniform outputs), which was then fixed (trajectory#182, #190, #192, #198). The final ensemble ens(v5l+v4l) was then chosen by comparing candidates on the train-split dev data (trajectory#238, #239). validate.py was not run on the final ensemble, and its score did not feed training, calibration or selection.
✓no jev reproductionsThe only base models downloaded were microsoft/deberta-v3-base, answerdotai/ModernBERT-base and answerdotai/ModernBERT-l
The only base models downloaded were microsoft/deberta-v3-base, answerdotai/ModernBERT-base and answerdotai/ModernBERT-large, all general pretrained encoders (trajectory#102 onward, snapshot_download calls). The only teacher was deepseek-chat-v3-0324 via Spin. No JEV-reproduction model, checkpoint, dataset or teacher output appears in the code, the manifests or the trajectory.
✓stateless zero shot inferenceserve.py scores each request independently. It encodes the text and that request's own label descriptions, applies a sof
serve.py scores each request independently. It encodes the text and that request's own label descriptions, applies a softmax over those labels only, and averages the two models' outputs. It has no cross-request caches, no label or dataset lookup tables, no external network calls and no online adaptation. Warmup uses fixed strings, and the fallback on error is a uniform distribution. serve.sh installs only the vendored wheels offline, then execs serve.py.
✓actual inference uses reviewed weightsserve.sh runs serve.py with --model-dir $SUB/model and --model-dir2 $SUB/model2. Each Encoder builds the architecture fr
serve.sh runs serve.py with --model-dir $SUB/model and --model-dir2 $SUB/model2. Each Encoder builds the architecture from config (AutoModel.from_config plus BiEncoderClassifier) and runs load_state_dict(load_file(model.safetensors)). The shipped files are model/model.safetensors and model2/model.safetensors. Their inventories show the ModernBERT-large backbone plus the trained head tensors, and their sha256 values are recorded in training_manifest.json.
✓harness credentials used only by the harnessThe LLM key used for synthetic data generation came from `spin openrouter key create` (trajectory#102) and was stored in
The LLM key used for synthetic data generation came from `spin openrouter key create` (trajectory#102) and was stored in /workspace/artifacts/openrouter.key. gen_synth.py and gen_synth2.py read it from there and call the Spin proxy, which is allowed. The trajectory shows no reads of the harness's OPENROUTER or ANTHROPIC environment credentials, no printenv, and no claude or codex CLI launches.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 27m into the runtook 3m 18s
startup 71.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 71.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.003342358661602871, 0.0033885564559340262, 0.9482944170750423, 0.007981043245996532, 0.003632131903942002, 0.003787070115836798, 0.004922256032799269, 0.0034645842818747797, 0.003283464494846961, 0.003486594113617946, 0.0036529955103 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 26.7 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.19, p95 0.24
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed6h 7m into the runtook 3m 53s
startup 83.9 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 83.9 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0033833517827724104, 0.0030822079140726878, 0.9518021543518832, 0.007881819099543495, 0.003506254005572036, 0.0030424886340083214, 0.00422348127375818, 0.0032305812285398586, 0.002971513701048463, 0.003697013093189088, 0.003438810997 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 35.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.26, p95 0.29
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
All checks pass. Stopping the GPU rental and submitting: