Zero-shot classification
- Score
- 13.8
- Wallet spend
- $4.12 of $10
- Model cost
- $14.75
- Agent time
- 3h 10m
- Steps
- 204
- Tokens in / out
- 19.4M / 165k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 11th of 14
Score
13.8raw metric 0.385
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.470 | 0.223 | 1.00 | 31.7 |
| amazonpolarity | 0.498 | 0.522 | 1.00 | 0.0 |
| appreviews | 0.638 | 0.521 | 1.00 | 24.4 |
| banking77 | 0.0436 | 0.0142 | 1.00 | 3.0 |
| biasframes_intent | 0.455 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.472 | 0.456 | 1.00 | 2.9 |
| biasframes_sex | 0.490 | 0.369 | 1.00 | 19.2 |
| capsotu | 0.0426 | 0.0166 | 1.00 | 2.6 |
| emotiondair | 0.198 | 0.132 | 1.00 | 7.5 |
| empathetic | 0.0379 | 0.0300 | 1.00 | 0.8 |
| financialphrasebank | 0.656 | 0.300 | 1.00 | 50.9 |
| imdb | 0.337 | 0.510 | 1.00 | 0.0 |
| manifesto | 0.00692 | 0.00957 | 1.00 | 0.0 |
| massive | 0.143 | 0.0199 | 1.00 | 12.5 |
| rottentomatoes | 0.415 | 0.509 | 1.00 | 0.0 |
| trueteacher | 0.333 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.740 | 0.511 | 1.00 | 46.7 |
| wikitoxic_obscene | 0.760 | 0.514 | 1.00 | 50.6 |
| wikitoxic_threat | 0.529 | 0.360 | 1.00 | 26.4 |
| wikitoxic_toxicaggregated | 0.614 | 0.508 | 1.00 | 21.5 |
| yahootopics | 0.0872 | 0.0647 | 1.00 | 2.4 |
| yelpreviews | 0.505 | 0.513 | 1.00 | 0.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.4242
- ece
- 0.0943
- accuracy
- 0.4630
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the task, and the record holds together from end to end. train.py fully fine-tunes stock Qwen2.
Training happened during the task, and the record holds together from end to end. train.py fully fine-tunes stock Qwen2.5-0.5B (revision 060db649) as a listwise set classifier with a new MLP head and two added tokens. training.log records 17,254 steps, with loss falling from 3.94 to about 0.33 and the best dev checkpoint at step 16,500. export.py writes model/model.safetensors. The weight inventory matches that: new head1/head2 tensors and embeddings expanded to 151,938 rows. The trajectory shows the training run on the rented L40S box and a hash comparison before the bundle was copied out.
✓no evaluation data useEvery training source declared in training_manifest.json and training/data_manifest.json is a train split of an allowed
Every training source declared in training_manifest.json and training/data_manifest.json is a train split of an allowed corpus. Each is listed with a sha256, though no retained download log was found, so the hashes are unverified. The manifests also declare no teacher models and no reused artifacts. prep_data.py reads validation.jsonl only to drop overlapping texts from train and dev. At trajectory#17 the participant queried the HF API for metadata on excluded datasets (for example SetFit/sst2 and amazon_polarity). No download was found in the trajectory, and none of them is in the shipped data, so this is not a violation.
✓no public split tuningThe temperature (1.75) and the best checkpoint were both chosen on the devbench dev sets, which are held-out training da
The temperature (1.75) and the best checkpoint were both chosen on the devbench dev sets, which are held-out training data (eval_dev.py, eval_dev.log), not on the public validation rows. validation.jsonl was used only for deduplication and for a final functional run of validate.py after export. The trajectory shows no checkpoint or hyperparameter being picked by validation score.
✓no jev reproductionsThe base is the stock Qwen/Qwen2.5-0.5B pinned to a revision, and training_manifest.json lists no teacher models or reus
The base is the stock Qwen/Qwen2.5-0.5B pinned to a revision, and training_manifest.json lists no teacher models or reused artifacts. Jeff, Laya and Open-Jev appear in the trajectory only as policy text, never as something the participant downloaded or used. The architecture (qwen_model.py, SetClassifier) is a fresh fine-tune.
✓stateless zero shot inferenceserver.py scores each request on its own text and labels in a single forward pass. Dynamic batching only groups requests
server.py scores each request on its own text and labels in a single forward pass. Dynamic batching only groups requests together; there is no cache, memory or state carried from one request to another, and the warmup uses synthetic inputs.
✓actual inference uses reviewed weightsserve.sh starts server.py with MODEL_DIR set to the bundled model directory. server.py loads model/model.safetensors wit
serve.sh starts server.py with MODEL_DIR set to the bundled model directory. server.py loads model/model.safetensors with strict=True into the SetClassifier from qwen_model.py, using model/tokenizer.json and model/serving_config.json. The reviewed weight inventory (embed.weight [151938, 896], head1/head2 and the standard Qwen layers) matches that module exactly.
✓harness credentials used only by the harnessHF_TOKEN, which the task prompt describes as a read-only token for downloading models and datasets, was used only to fet
HF_TOKEN, which the task prompt describes as a read-only token for downloading models and datasets, was used only to fetch the Qwen base and the training datasets. Compute was rented through the sanctioned spin CLI. The per-rental SSH key returned by `spin compute ssh` was used only to reach the participant's own rental. SPIN_BROKER_TOKEN, SPIN_API_KEY and MLE_JUDGE_API_KEY appear only in docs and help output and were never read, exported or used by the participant.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 7m into the runtook 1m 57s
startup 47.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 47.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0010386880703530195, 0.0010386880703530195, 0.9874301094401446, 0.0010386880703530195, 0.0010386880703530195, 0.0010386880703530195, 0.0010386880703530195, 0.0010386880703530195, 0.0010386880703530195, 0.00010563371561916497, 0.00103 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 9.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.07, p95 0.08
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. ✅
Final summary
Model — Bidirectional (LLM2Vec-style) adaptation of Qwen2.5-0.5B as a listwise set classifier: one sequence holds <|zsc_txt|> + text + every candidate label after its own <|zsc_lbl|> marker, full non-causal attention, with a 2-layer MLP head over [h_label ; h_text ; h_label·h_text] producing per-label scores. Full-parameter fine-tune (17,254 steps, 2 epochs) on 653,190 examples from 26 allowed training-split corpora — nothing from the BTZSC exclusion list, with public-validation texts filtered out.
Results
| Metric | Value |
|---|---|
| Devbench macro-F1 (held-out, 3 label phrasings) | 0.689 |
| Devbench ECE (temp 1.75, fitted on held-out rows only) | 0.068 |
| Devbench score (F1 × (1−ECE)) | 0.645 |
| Public diagnostic (500 DBpedia rows, functional check only) | 0.961 |
| Selfcheck on real H100: throughput | 478.7 docs/s (gate: 50) |
| Startup | 47.4 s (gate: 300) |
| Failed/malformed responses | 0 of 4400 |
Submission (/workspace/output, 63 files, 4.93 GB): serve.sh + server.py + qwen_model.py, model/ (bf16 safetensors + tokenizer + calibration config), vendored PyPI wheels for offline install, full training code/logs/config, training_manifest.json (26 sources with per-file sha256 + repo revisions), architecture.json, and data provenance. All evidence paths verified in place, no symlinks, and the selfcheck's throughput margin (857%) comfortably covers the ~1.75× larger held-out requests.
Key fixes along the way: config key mismatch on serving reload, accept-backlog connection resets under 32-way bursts, and an eval target-index misalignment after label shuffling — each caught and verified before the final run.