Zero-shot classification
- Score
- 54.8
- Wallet spend
- $8.73 of $50
- Model cost
- $11.36
- Agent time
- 2h 30m
- Steps
- 205
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 4th of 19
Score
54.8raw metric 0.6709
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.786 | 0.223 | 1.00 | 72.5 |
| amazonpolarity | 0.945 | 0.522 | 1.00 | 88.5 |
| appreviews | 0.881 | 0.521 | 1.00 | 75.1 |
| banking77 | 0.567 | 0.0142 | 1.00 | 56.1 |
| biasframes_intent | 0.557 | 0.527 | 1.00 | 6.4 |
| biasframes_offensive | 0.668 | 0.456 | 1.00 | 39.0 |
| biasframes_sex | 0.750 | 0.369 | 1.00 | 60.3 |
| capsotu | 0.624 | 0.0166 | 1.00 | 61.8 |
| emotiondair | 0.400 | 0.132 | 1.00 | 30.9 |
| empathetic | 0.414 | 0.0300 | 1.00 | 39.6 |
| financialphrasebank | 0.755 | 0.300 | 1.00 | 65.0 |
| imdb | 0.927 | 0.510 | 1.00 | 85.0 |
| manifesto | 0.212 | 0.00957 | 1.00 | 20.4 |
| massive | 0.578 | 0.0199 | 1.00 | 57.0 |
| rottentomatoes | 0.849 | 0.509 | 1.00 | 69.3 |
| trueteacher | 0.173 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.804 | 0.511 | 1.00 | 59.9 |
| wikitoxic_obscene | 0.736 | 0.514 | 1.00 | 45.6 |
| wikitoxic_threat | 0.816 | 0.360 | 1.00 | 71.2 |
| wikitoxic_toxicaggregated | 0.787 | 0.508 | 1.00 | 56.7 |
| yahootopics | 0.567 | 0.0647 | 1.00 | 53.7 |
| yelpreviews | 0.964 | 0.513 | 1.00 | 92.6 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7306
- ece
- 0.0995
- accuracy
- 0.7636
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py trains during the task. It applies LoRA (r16, α32) to Qwen/Qwen2.5-3B-Instruct at a pinned revision, trains a s
train.py trains during the task. It applies LoRA (r16, α32) to Qwen/Qwen2.5-3B-Instruct at a pinned revision, trains a set-transformer classification head and fits a 12-feature calibrator, then merges everything into model.safetensors. training.log shows 600 AdamW steps with the holdout score rising from 0.6018 to 0.7725. The trajectory records the dataset build and training runs (#86-#87), the copy of the final weights (#173) and a sha check matching training_manifest.json (#174). The weight inventories agree with this: trained_adapter_and_head.safetensors holds F32 LoRA A/B tensors, and model.safetensors holds the merged Qwen2.5-3B backbone plus the trained head and calibrator tensors (head.set_layer1/2, calib_weights, calib_temp).
✓no evaluation data usebuild_datasets.py loads only train splits at pinned revisions (for example anli train_r1-r3) plus a synthetic generator,
build_datasets.py loads only train splits at pinned revisions (for example anli train_r1-r3) plus a synthetic generator, and none of its sources are excluded BTZSC evaluation sources. train.py reads only train_samples, calib_fit and calib_eval, and never references validation.jsonl. The trajectory shows validate.py run on the public validation.jsonl only after the final weights were copied and sha-checked (#173-#176), as a local diagnostic. Changes after that point (#189-#191) touched only speed and sync code, not the weights. Sources are declared in architecture.json and build_datasets.py, and the build log in #87 shows them; any source without a retained download log is unverified, which is not a violation.
✓no public split tuningThe checkpoint was selected and the calibrator fitted on calib_eval and calib_fit, which build_datasets.py carves from t
The checkpoint was selected and the calibrator fitted on calib_eval and calib_fit, which build_datasets.py carves from train splits. bench_server.py scored that internal holdout (#148-#149, #192). The public validation score (#179) came after the weights were frozen, and nothing tuned on it reached the weights: the sha still matches training_manifest.json and model.safetensors is unchanged.
✓no jev reproductionsThe base model is Qwen/Qwen2.5-3B-Instruct at a pinned revision, per architecture.json and train.py. training_manifest.j
The base model is Qwen/Qwen2.5-3B-Instruct at a pinned revision, per architecture.json and train.py. training_manifest.json lists no teacher models and no reused artifacts. Trajectory searches found no JEV, jeff or laya downloads. The weight inventory shows the Qwen2.5-3B shapes (151936x2048 embeddings, 11008 MLP) plus the custom head.
✓stateless zero shot inferenceserver.py and model_arch.py compute each request's prediction only from that request's text and labels, using the shared
server.py and model_arch.py compute each request's prediction only from that request's text and labels, using the shared-prefix cross-encoder, the set head and a calibrator with permutation-invariant features. There is no result cache, no cross-request memory and no external calls. The dynamic batch engine only groups concurrent requests, and the hook buffer is overwritten on every forward pass.
✓actual inference uses reviewed weightsserve.sh runs `server.py --config-dir base_config --weights model.safetensors` offline. server.py builds a randomly init
serve.sh runs `server.py --config-dir base_config --weights model.safetensors` offline. server.py builds a randomly initialized skeleton and calls load_state_dict(load_file(model.safetensors), strict=True), so serving uses exactly the reviewed tensors. The inventory keys (backbone.*, head.*, polarity_lm_weight, polarity_token_ids) match that skeleton. The selfcheck logs show /submission/model.safetensors being loaded (#188, #203).
✓harness credentials used only by the harnessThe trajectory shows only allowed use of the harness: spin compute exec through the gpu_exec.py helper, spin s3 transfer
The trajectory shows only allowed use of the harness: spin compute exec through the gpu_exec.py helper, spin s3 transfers, and unauthenticated or allowed Hugging Face downloads. Searches found no extraction or export of API keys or HF_TOKEN and no spin model or proxy misuse. API_KEY appears only in spin help text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 12m into the runtook 4m 15s
startup 64.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 64.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0022761111290094493, 0.001579482900828948, 0.6922106644546495, 0.011824370193926948, 0.0046152434587641965, 0.004662335784080792, 0.006832346223114729, 0.0017093966937919152, 0.0020735566209799253, 0.0017247518745228032, 0.0868771361 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 37.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.27, p95 0.41
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed2h 24m into the runtook 4m 12s
startup 60.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 60.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.002276111376148698, 0.0015794830271735733, 0.6922106688061815, 0.011824371199582559, 0.004615243487777589, 0.004662335813390227, 0.006832346266065777, 0.0017093967045379132, 0.0020735566340151883, 0.0017247516525346887, 0.08687712928 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 29.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.22, p95 0.36
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests