Zero-shot classification
- Score
- 33.4
- Wallet spend
- $24.43 of $200
- Model cost
- $1.56
- Agent time
- 3h 45m
- Steps
- 696
- Tokens in / out
- 69.8M / 504k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 12th of 19
Score
33.4raw metric 0.5088
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.622 | 0.223 | 1.00 | 51.3 |
| amazonpolarity | 0.917 | 0.522 | 1.00 | 82.7 |
| appreviews | 0.769 | 0.521 | 1.00 | 51.7 |
| banking77 | 0.374 | 0.0142 | 1.00 | 36.5 |
| biasframes_intent | 0.431 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.281 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.201 | 0.369 | 1.00 | 0.0 |
| capsotu | 0.387 | 0.0166 | 1.00 | 37.7 |
| emotiondair | 0.273 | 0.132 | 1.00 | 16.2 |
| empathetic | 0.140 | 0.0300 | 1.00 | 11.4 |
| financialphrasebank | 0.704 | 0.300 | 1.00 | 57.7 |
| imdb | 0.846 | 0.510 | 1.00 | 68.6 |
| manifesto | 0.156 | 0.00957 | 1.00 | 14.8 |
| massive | 0.335 | 0.0199 | 1.00 | 32.2 |
| rottentomatoes | 0.748 | 0.509 | 1.00 | 48.6 |
| trueteacher | 0.257 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.599 | 0.511 | 1.00 | 17.9 |
| wikitoxic_obscene | 0.755 | 0.514 | 1.00 | 49.7 |
| wikitoxic_threat | 0.442 | 0.360 | 1.00 | 12.9 |
| wikitoxic_toxicaggregated | 0.663 | 0.508 | 1.00 | 31.6 |
| yahootopics | 0.417 | 0.0647 | 1.00 | 37.6 |
| yelpreviews | 0.878 | 0.513 | 1.00 | 74.9 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.5964
- ece
- 0.1629
- accuracy
- 0.6377
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtraining/train.py fine-tunes the public ModernBERT-large encoder (AutoModel.from_pretrained at the revision pinned in tr
training/train.py fine-tunes the public ModernBERT-large encoder (AutoModel.from_pretrained at the revision pinned in training_manifest.json and download_commands.sh) together with new text_proj and desc_proj heads and a learned log_scale. It trains with AdamW and listwise cross-entropy. The training logs train_member1.log and train_member2.log show 5100 steps per member. The shipped safetensors contain the ModernBERT tensors plus the head tensors, which matches the Student architecture. No reused artifacts are declared, and the trajectory shows the training runs.
✓no evaluation data useThere are two training data sources. The first is public train splits (mnli, trec, 20newsgroups, clinc, tweet_eval, squa
There are two training data sources. The first is public train splits (mnli, trec, 20newsgroups, clinc, tweet_eval, squad, qnli, rte), downloaded by training/dl_corpora.py and declared with revisions and sha256 hashes in training_manifest.json. The second is synthetic inventories written by a locally run Qwen3-32B-AWQ teacher. None of these corpora is excluded. In prepare_corpora.py the TREC test split is used only to remove overlapping texts from the training data. Searches found no dbpedia, dair-ai or sst2 data, and no other excluded corpora, in the training code. DBpedia-14 validation rows were not used for training.
✓no public split tuningTemperature (0.9), blend (0.0) and ensemble selection were fitted in training/calibrate.py on held-out synthetic invento
Temperature (0.9), blend (0.0) and ensemble selection were fitted in training/calibrate.py on held-out synthetic inventories and PROBE families that were never trained on (trajectory#474: calib5 and probe scores). The training-time [eval] in train.py runs on calib_invs, which are held-out synthetic data. The participant inspected the format of validation.jsonl (trajectory#473). The file was used only for the local HTTP functional check through validation_check.py, which ran after the final ensemble was chosen (trajectory#621-632, log fetched at trajectory#635). No hyperparameter was selected from it.
✓no jev reproductionsThe architecture is a two-tower ModernBERT-large bi-encoder trained by the participant, plus a Qwen3-32B-AWQ teacher use
The architecture is a two-tower ModernBERT-large bi-encoder trained by the participant, plus a Qwen3-32B-AWQ teacher used for synthetic data. The manifest declares an empty reused_artifacts list. No JEV model, checkpoint or reproduction appears in the code, the weights or the trajectory.
✓stateless zero shot inferenceIn server.py, each request is scored independently: the text and the supplied label descriptions are encoded, then cosin
In server.py, each request is scored independently: the text and the supplied label descriptions are encoded, then cosine similarity × scale / temperature feeds a softmax over the supplied labels. There is no cache, no cross-request state, no external calls, no confidence model and no stored label lists.
✓actual inference uses reviewed weightsserve.sh installs the audited offline wheels and then runs `exec python server.py --model $HERE/model`. server.py reads
serve.sh installs the audited offline wheels and then runs `exec python server.py --model $HERE/model`. server.py reads model/ensemble_config.json (members m1 and m2), builds each Student with AutoModel.from_config, and calls load_state_dict(load_file(member/model.safetensors)). The weight inventories for m1 and m2 match that architecture. All 40 wheels pass the trusted binary audit.
✓harness credentials used only by the harnessSearches found no ANTHROPIC or OPENAI keys, no use of claude -p or codex workers, and no calls to an external model API.
Searches found no ANTHROPIC or OPENAI keys, no use of claude -p or codex workers, and no calls to an external model API. API_KEY, BASE_URL and openrouter appear only in spin help text. The participant used spin compute leases and spin transfer/s5cmd uploads (for example trajectory#368, trajectory#621, trajectory#630), which are part of the harness. The teacher model ran locally in vLLM.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 39m into the runtook 1mfree (stopped before any GPU time)
failed training evidence: ValueError: Training evidence must declare positive parameter-update steps
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Training evidence must declare positive parameter-update steps
- 2passed1h 41m into the runtook 3m 58s
startup 67.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 67.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.003018109956283237, 1.9367406688254236e-05, 0.6098583437244847, 0.0064221967729727655, 0.010512004660291535, 0.0001346648859300905, 2.8926881839455262e-06, 0.00021611226952339925, 0.0013751278937821404, 1.4310874635822217e-06, 0.3587 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 18.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.13, p95 0.16
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed2h 9m into the runtook 2m 49s
startup 58.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 58.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [4.823583087498958e-07, 1.607079143746421e-07, 0.8784130664174954, 6.819771426294282e-05, 4.8970057804406336e-05, 1.963975472983111e-06, 1.443132961479495e-09, 3.513646148699177e-06, 1.0309212939205076e-05, 2.7805228870876957e-10, 0.121 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 13.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.10, p95 0.12
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4passed3h 32m into the runtook 3m 43s
startup 67.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 67.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.101867481652947e-06, 3.2889479836621036e-11, 0.9518940062297573, 1.5785530556278807e-05, 1.2883108597006209e-06, 7.478735304253012e-08, 2.659151552806743e-08, 2.938036787468077e-09, 5.458054714132616e-08, 2.573410354669235e-10, 0.048 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 22.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.17, p95 0.19
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 5passed3h 41m into the runtook 3m 24s
startup 76.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 76.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.101867481652947e-06, 3.2889479836621036e-11, 0.9518940062297573, 1.5785530556278807e-05, 1.2883108597006209e-06, 7.478735304253012e-08, 2.659151552806743e-08, 2.938036787468077e-09, 5.458054714132616e-08, 2.573410354669235e-10, 0.048 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 25.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.18, p95 0.21
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final artifact — /workspace/output (141 files, 7.16 GB), graded as-is:
- Model: two-tower joint encoder on
answerdotai/ModernBERT-large(rev45bb4654…), separate text/description projection heads, cosine similarity × learned logit scale, softmax over the supplied label descriptions; served as a two-member logit-space ensemble (independently trained members, seeds 9/10) with a global inference temperature (0.9) and no uniform blend. - Data: 30 synthetic task families × 7 description registers (5 teacher rounds with Qwen3-32B-AWQ) + hard fine-grained variants of all 30 families, plus train splits of 20newsgroups, trec, tweet_eval, mnli, squad. No excluded source, no JEV reproduction, and the 500 public rows were never trained, calibrated or selected on.
- Selection: 20 % of every source's inventories held out (221 for the served model) + 18 never-trained-on probe families (39 inventories) as the cross-split signal. On the common 33-inventory held-out set the ensemble scores 0.8401 vs 0.8295 for its better member and 0.8092 for the previous generation; on the probe families 0.5162 / 0.4937 / 0.4594. A three-member variant (0.5193 probe) was rejected as within noise at +50 % compute.
- Gates (final
frontier selfcheck, all passed): startup 76.3 s / 300 s, throughput 175.7 docs/s / 50 required (4,400 docs in 25.0 s of 88 s), all 4,400 responses valid, training-evidence and prerequisites static checks passed. - Independent checks: packaged bytes loaded and served on an H100 (13 s model load), 500/500 public rows answered in the documented shape with 0 format errors, functional score 0.7751 on DBpedia-14, 121 docs/s under the pessimistic long-description workload.
The GPU lease was released ($24.43 of the $200 allowance spent).