Zero-shot classification
- Score
- 52.5
- Wallet spend
- $5.24 of $10
- Model cost
- $20.45
- Agent time
- 39m
- Steps
- 110
- Tokens in / out
- 14.3M / 76k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 4th of 14
Score
52.5raw metric 0.6628
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.836 | 0.223 | 1.00 | 78.9 |
| amazonpolarity | 0.899 | 0.522 | 1.00 | 78.9 |
| appreviews | 0.820 | 0.521 | 1.00 | 62.4 |
| banking77 | 0.495 | 0.0142 | 1.00 | 48.8 |
| biasframes_intent | 0.770 | 0.527 | 1.00 | 51.3 |
| biasframes_offensive | 0.698 | 0.456 | 1.00 | 44.5 |
| biasframes_sex | 0.581 | 0.369 | 1.00 | 33.6 |
| capsotu | 0.569 | 0.0166 | 1.00 | 56.2 |
| emotiondair | 0.376 | 0.132 | 1.00 | 28.0 |
| empathetic | 0.458 | 0.0300 | 1.00 | 44.2 |
| financialphrasebank | 0.711 | 0.300 | 1.00 | 58.6 |
| imdb | 0.911 | 0.510 | 1.00 | 81.9 |
| manifesto | 0.242 | 0.00957 | 1.00 | 23.5 |
| massive | 0.601 | 0.0199 | 1.00 | 59.3 |
| rottentomatoes | 0.829 | 0.509 | 1.00 | 65.1 |
| trueteacher | 0.353 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.758 | 0.511 | 1.00 | 50.4 |
| wikitoxic_obscene | 0.816 | 0.514 | 1.00 | 62.1 |
| wikitoxic_threat | 0.577 | 0.360 | 1.00 | 34.0 |
| wikitoxic_toxicaggregated | 0.735 | 0.508 | 1.00 | 46.2 |
| yahootopics | 0.592 | 0.0647 | 1.00 | 56.4 |
| yelpreviews | 0.953 | 0.513 | 1.00 | 90.2 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7249
- ece
- 0.0965
- accuracy
- 0.7659
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py loads the pinned base model Qwen/Qwen3-4B-Instruct-2507 (revision cdbee75f…, fetched with snapshot_download). I
train.py loads the pinned base model Qwen/Qwen3-4B-Instruct-2507 (revision cdbee75f…, fetched with snapshot_download). It wraps the model in a fresh rank-8 LoRA via get_peft_model on q/v/o_proj and trains for 300 AdamW steps with a cross-entropy loss over code tokens. The H100 run is logged in evidence/training.log: 4,866,048 trainable parameters, loss curve shown, checkpoints at steps 75, 150 and 300. The trajectory shows the run on a leased GPU and the trained adapter copied into adapter/. The inventory of adapter/adapter_model.safetensors holds LoRA A/B tensors only, consistent with an adapter trained during the task. No external fine-tuned checkpoint is used.
✓no evaluation data useprepare_data.py downloads only files that are train splits (or README) from pinned Hugging Face revisions: CLINC150 plus
prepare_data.py downloads only files that are train splits (or README) from pinned Hugging Face revisions: CLINC150 plus, GoEmotions simplified, TweetEval hate/offensive/irony/stance, MultiNLI and the SetFit 20 Newsgroups and TREC-QC train files. Every source is declared in architecture.json and training_manifest.json; the download logs are retained. None is a BTZSC evaluation source or derivative. TweetEval sentiment was fetched in an early attempt, then removed before training (train.py asserts its absence), and it is not a BTZSC source anyway. Social Bias Frames was only looked up in web documentation to confirm provenance; none of its data was downloaded. No teacher models are used.
✓no public split tuningSelection and calibration partitions are disjoint slices of the training files (i%10 split), recorded with content hashe
Selection and calibration partitions are disjoint slices of the training files (i%10 split), recorded with content hashes. train.py chooses the checkpoint and fits the temperature on those partitions alone. The public validator (/task/validate.py) ran once, at trajectory step 95. That was after the step-300 checkpoint was chosen on the training-partition score and after the model was frozen with recorded hashes (step 93). Hashes were re-verified unchanged afterwards (step 99), and no model, prompt or calibration change followed.
✓no jev reproductionsThe data sources are general public training corpora with authored label descriptions produced deterministically in prep
The data sources are general public training corpora with authored label descriptions produced deterministically in prepare_data.py. No JEV reproduction artifacts, teacher outputs or benchmark-derived predictions appear in the code, manifests or trajectory.
✓stateless zero shot inferenceserver.py builds each prompt from the request's own text and labels through modeling.PromptEncoder. It runs a single for
server.py builds each prompt from the request's own text and labels through modeling.PromptEncoder. It runs a single forward pass and applies a softmax restricted to the request's labels, with a fixed temperature from inference_config.json. Labels are sorted canonically, which makes the output permutation-equivariant. Batching groups concurrent requests only for throughput. There is no cache across requests, no per-request adaptation and no stored state.
✓actual inference uses reviewed weightsserve.sh installs the local wheels offline and runs server.py. server.py calls modeling.load_lm(ROOT/base_model, ROOT/ad
serve.sh installs the local wheels offline and runs server.py. server.py calls modeling.load_lm(ROOT/base_model, ROOT/adapter): AutoModelForCausalLM with local_files_only, then PeftModel.from_pretrained(adapter).merge_and_unload(). ChoiceModel scores with the lm_head rows of the code tokens. The adapter shipped is the inventoried adapter/adapter_model.safetensors (sha256 6ed29cfe…, matching training_manifest.json and model_freeze.json), copied from the training output at trajectory step 91. The selfcheck server log shows the step-300 adapter and temperature being loaded.
✓harness credentials used only by the harnessThe trajectory shows only allowed tooling: spin compute/s5cmd/openrouter model listing, Hugging Face downloads and front
The trajectory shows only allowed tooling: spin compute/s5cmd/openrouter model listing, Hugging Face downloads and frontier selfcheck/submit. Searches found no direct model-API calls with harness keys, no copying of API keys and no launch of claude or codex CLIs.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed34m 21s into the runtook 4m 6s
startup 59.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 59.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.002644302769250963, 0.0017611796822588164, 0.7823670767140839, 0.003970250093782761, 0.003970250093782761, 0.002644302769250963, 0.000894591472490757, 0.0023092718734836677, 0.001172994979723854, 0.000894591472490757, 0.0201765199344 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 37.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.27, p95 0.33
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Self-check passed: 59.5-second offline startup, 118.4 documents/second, and all 4,400 responses valid. Training and calibration used only permitted training data.