Zero-shot classification
- Score
- 52.3
- Wallet spend
- $5.88 of $10
- Model cost
- $6.81
- Agent time
- 3h 9m
- Steps
- 272
- Tokens in / out
- 108.8M / 457k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 5th of 14
Score
52.3raw metric 0.6595
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.825 | 0.223 | 1.00 | 77.5 |
| amazonpolarity | 0.920 | 0.522 | 1.00 | 83.3 |
| appreviews | 0.844 | 0.521 | 1.00 | 67.3 |
| banking77 | 0.669 | 0.0142 | 1.00 | 66.4 |
| biasframes_intent | 0.438 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.536 | 0.456 | 1.00 | 14.7 |
| biasframes_sex | 0.707 | 0.369 | 1.00 | 53.6 |
| capsotu | 0.642 | 0.0166 | 1.00 | 63.6 |
| emotiondair | 0.454 | 0.132 | 1.00 | 37.1 |
| empathetic | 0.453 | 0.0300 | 1.00 | 43.7 |
| financialphrasebank | 0.799 | 0.300 | 1.00 | 71.2 |
| imdb | 0.904 | 0.510 | 1.00 | 80.4 |
| manifesto | 0.235 | 0.00957 | 1.00 | 22.7 |
| massive | 0.594 | 0.0199 | 1.00 | 58.5 |
| rottentomatoes | 0.788 | 0.509 | 1.00 | 56.7 |
| trueteacher | 0.184 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.807 | 0.511 | 1.00 | 60.6 |
| wikitoxic_obscene | 0.806 | 0.514 | 1.00 | 60.0 |
| wikitoxic_threat | 0.588 | 0.360 | 1.00 | 35.6 |
| wikitoxic_toxicaggregated | 0.768 | 0.508 | 1.00 | 52.9 |
| yahootopics | 0.616 | 0.0647 | 1.00 | 58.9 |
| yelpreviews | 0.933 | 0.513 | 1.00 | 86.1 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7186
- ece
- 0.0987
- accuracy
- 0.7616
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTraining happened during the task session. train.py starts from the pinned Qwen3-1.7B-Base weights (rev ea980cb0…), adds
Training happened during the task session. train.py starts from the pinned Qwen3-1.7B-Base weights (rev ea980cb0…), adds LoRA (r32/α64) and a fresh linear pointer head, and trains them with AdamW and cross-entropy. The trajectory shows the F17 run on the H100: 1733 steps, 760 s, loss going down, and the adapter uploaded. package_model.py then merged the adapter into model.safetensors (sha256 414691b0…), which the manifest records. The shipped adapter and checkpoint both contain the trained head tensors. reused_artifacts is empty.
✓no evaluation data useAll training corpora are the train splits of 10 declared public datasets, pinned by SHA. fetch_public.sh and its log ret
All training corpora are the train splits of 10 declared public datasets, pinned by SHA. fetch_public.sh and its log retain the download commands. The rest of the data is a declared synthetic corpus written by qwen3-30b-a3b-instruct-2507 through the spin OpenRouter proxy, with a prompt that forbids reproducing datasets. build_data.py never reads the excluded BTZSC sources. validation.jsonl (DBpedia test) was used only for validate.py contract checks and throughput timing. It was never used for training data, calibration or scoring of training rows.
✓no public split tuningT=1.3 was chosen from held-out training-split cal/dev rows. Dev runs A17/A4B held out the clinc, ng20 and tw_emotion tra
T=1.3 was chosen from held-out training-split cal/dev rows. Dev runs A17/A4B held out the clinc, ng20 and tw_emotion train splits; fitted OOD T was 1.42–1.69 and in-distribution T 1.0–1.1 (trajectory#121). The choice between 4B and 1.7B was made on dev accuracy and server throughput, not on validation accuracy (trajectory#219, trajectory#221). Validation macro-F1 was viewed only as a server contract check (trajectory#159) and was not used to select the model, prompt or temperature.
✓no jev reproductionsJeff/Laya/Open-Jev/JEV-4B/typesafe-diffusion-lab are mentioned only in the policy text (trajectory#1–10). No download, c
Jeff/Laya/Open-Jev/JEV-4B/typesafe-diffusion-lab are mentioned only in the policy text (trajectory#1–10). No download, code or artifact of theirs appears. The model is an independent Qwen3-1.7B-Base LoRA with its own qwen3_min implementation and pointer head, and the manifest lists no reused artifacts.
✓stateless zero shot inferenceserve.py tokenizes each request's text and labels with a fixed prompt template and scores them all in one forward pass.
serve.py tokenizes each request's text and labels with a fixed prompt template and scores them all in one forward pass. It applies a linear head at each label's newline token, then softmax(logits/T). There is no cache, retrieval, cross-request state, external call or label-specific memory. Dynamic batching only groups the requests that are currently queued.
✓actual inference uses reviewed weightsserve.sh runs serve/serve.py. That calls engine.load_model(model/), which does load_file(model/model.safetensors) and th
serve.sh runs serve/serve.py. That calls engine.load_model(model/), which does load_file(model/model.safetensors) and then load_state_dict(strict=True) into the qwen3_min PointerClassifier. The weight inventory shows the body plus the head.weight/head.bias tensors. The trajectory shows the F17 package (sha 414691b0…) copied into output/model and its logits checked against GPU-scored logits (trajectory#255, trajectory#256).
✓harness credentials used only by the harnessThe participant's teacher calls used a scoped key from `spin openrouter key create` stored in /tmp/or_key.json (trajecto
The participant's teacher calls used a scoped key from `spin openrouter key create` stored in /tmp/or_key.json (trajectory#11–26), which the policy allows. No ANTHROPIC/OPENAI keys or *_BASE_URL harness credentials appear in the code or the trajectory. No extra claude/codex/gemini CLI sessions were launched. HF downloads use HF_TOKEN, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 53m into the runtook 5m 24s
startup 61.9 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 61.9 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0006521690603824101, 0.0004534980435389119, 0.9302477863771679, 0.004275387815950343, 0.0006126672807455895, 0.0026649962391882337, 0.0018573431492856299, 0.0003321770166793242, 0.00011311581141882198, 0.001210326829884312, 0.0075871 ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 2passed3h 1m into the runtook 4m 4s
startup 108.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 108.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [3.66257424381033e-05, 1.9425241402261338e-05, 0.9725430425566838, 0.001443139511488233, 0.00026751692326149957, 0.000535526758185918, 7.441392837887575e-05, 4.562406654624032e-05, 2.5500380251923636e-06, 7.524651412238608e-05, 0.002735 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 19.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.14, p95 0.15
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed3h 6m into the runtook 3m 47s
startup 110.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 110.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [3.66257424381033e-05, 1.9425241402261338e-05, 0.9725430425566838, 0.001443139511488233, 0.00026751692326149957, 0.000535526758185918, 7.441392837887575e-05, 4.562406654624032e-05, 2.5500380251923636e-06, 7.524651412238608e-05, 0.002735 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 19.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.14, p95 0.16
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. frontier submit --confirm went through at 09:18 UTC, and the output folder is now graded as it stands.
What was submitted: Qwen3-1.7B-Base with a LoRA adapter (rank 32) and a per-label pointer head. The head reads each candidate label's score from the hidden state at the end of its line, so one forward pass handles any label set. Training used all allowed training splits: ten public corpora plus 1,157 synthetic tasks generated by a hosted teacher. Temperature is 1.3, a compromise between the in-distribution fit (about 1.0 to 1.1) and the held-out label sets (about 1.4 to 1.7).
Self-check (run 3 of 4, on the final folder, H100 SXM):
- Startup: 110.7 s, against a 300 s limit.
- Throughput: 222 documents per second, against 50 required. Held-out requests are about 1.75 times larger, which still leaves roughly 2 times margin.
- All 4,400 responses parsed, with no failed requests.
I did not get a score: the self-check reports no score, and the architecture review did not run. Its result is the one piece I could not verify, and it can still score the submission zero.
Choices I made that you may want to revisit:
- 1.7B over 4B. The 4B dev run was about one point more accurate, but its serving throughput was too close to the gate once held-out requests are heavier. I shipped 1.7B.
- Temperature 1.3. Chosen from dev-run calibration on held-out label sets, not from the public validation file.
- Serving changes. The first server was far too slow on the grader. Two fixes brought it up: bucketing padded shapes so the GPU doesn't recompile for each new one, and CUDA graphs captured at startup, with an eager fallback if capture fails.
Caveats:
- The macro-F1 figures in the dev logs are approximate because classes were keyed by label string. Accuracy, NLL and ECE are exact on the held-out rows.
- The submitted 1.7B build was checked for response format and speed, not scored on the public validation file.
- The synthetic corpus is kept outside the submission, with its digest recorded in the manifest.
- Spend was about $5.3 of the $10 budget.
The README in /workspace/output lists the layout, method, dev numbers, and serving measurements.