Zero-shot classification
- Score
- 56.5
- Wallet spend
- $30.50 of $200
- Model cost
- $5.07
- Agent time
- 2h 46m
- Steps
- 129
- Tokens in / out
- 17.2M / 172k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 3rd of 14
Score
56.5raw metric 0.6759
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.786 | 0.223 | 1.00 | 72.4 |
| amazonpolarity | 0.954 | 0.522 | 1.00 | 90.3 |
| appreviews | 0.880 | 0.521 | 1.00 | 75.0 |
| banking77 | 0.537 | 0.0142 | 1.00 | 53.0 |
| biasframes_intent | 0.739 | 0.527 | 1.00 | 44.8 |
| biasframes_offensive | 0.706 | 0.456 | 1.00 | 45.9 |
| biasframes_sex | 0.633 | 0.369 | 1.00 | 41.8 |
| capsotu | 0.533 | 0.0166 | 1.00 | 52.5 |
| emotiondair | 0.367 | 0.132 | 1.00 | 27.0 |
| empathetic | 0.430 | 0.0300 | 1.00 | 41.2 |
| financialphrasebank | 0.785 | 0.300 | 1.00 | 69.3 |
| imdb | 0.944 | 0.510 | 1.00 | 88.6 |
| manifesto | 0.229 | 0.00957 | 1.00 | 22.2 |
| massive | 0.586 | 0.0199 | 1.00 | 57.7 |
| rottentomatoes | 0.830 | 0.509 | 1.00 | 65.3 |
| trueteacher | 0.251 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.851 | 0.511 | 1.00 | 69.5 |
| wikitoxic_obscene | 0.900 | 0.514 | 1.00 | 79.4 |
| wikitoxic_threat | 0.629 | 0.360 | 1.00 | 42.0 |
| wikitoxic_toxicaggregated | 0.829 | 0.508 | 1.00 | 65.3 |
| yahootopics | 0.511 | 0.0647 | 1.00 | 47.7 |
| yelpreviews | 0.961 | 0.513 | 1.00 | 92.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7520
- ece
- 0.1218
- accuracy
- 0.7973
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py trains a peft LoRA adapter (r16, alpha32, about 33M trainable parameters) on the downloaded and synthetic train
train.py trains a peft LoRA adapter (r16, alpha32, about 33M trainable parameters) on the downloaded and synthetic training data. The session shows both runs: - Phase 1 was launched (trajectory#59) and completed after 1,983 steps with checkpoint sha acfa83d9… and parameter_probe_max_change 0.0090 (trajectory#92). - Phase 2 was initialised from the phase-1 checkpoint via --init-adapter (trajectory#94) and completed with checkpoint sha 615399a3… and probe change 0.0028 (trajectory#117). The shipped adapter sha matches the phase-2 checkpoint recorded in training_manifest.json (trajectory#123).
✓no evaluation data usecollect_training.py downloads only train-*.parquet files at pinned revisions from four sources: clinc_oos plus, go_emoti
collect_training.py downloads only train-*.parquet files at pinned revisions from four sources: clinc_oos plus, go_emotions simplified, tweet_eval task configs and multi_nli. None of these is on the excluded list. Synthetic data was generated fresh by gpt-4.1-mini from authored task themes (generate_synthetic.py, trajectory#28, trajectory#54). All sources are declared in architecture.json. No excluded BTZSC, SST/MR/subj or CNN/DM-derived source appears anywhere in the code or the session.
✓no public split tuningCalibration and checkpoint selection (evaluate_training.py) use only selection.jsonl, which holds rows reserved from the
Calibration and checkpoint selection (evaluate_training.py) use only selection.jsonl, which holds rows reserved from the training splits plus independently generated synthetic tasks. The phase1/phase2/average comparison picked phase2 on that training holdout (0.7433 vs 0.7414 vs 0.7315). The model was frozen and quality_freeze.json was written before /task/validate.py ran (trajectory#122). The public validator was run afterwards only as an interface and throughput check, and the model did not change after it (trajectory#123, trajectory#124). Self-checks only exercised the server.
✓no jev reproductionsThe base model is Qwen3-4B-Instruct-2507, downloaded from its official repository at a pinned sha (trajectory#14). The t
The base model is Qwen3-4B-Instruct-2507, downloaded from its official repository at a pinned sha (trajectory#14). The tensor inventories for base/ show standard Qwen shards. The adapter is a LoRA delta trained during the session. No JEV reproduction checkpoint or artifact was obtained or shipped.
✓stateless zero shot inferenceserver.py handles each /classify request independently. It encodes the text and labels with the chat-template prompt, ru
server.py handles each /classify request independently. It encodes the text and labels with the chat-template prompt, runs a single forward pass, projects the last hidden state onto lm_head rows for the choice tokens, applies a fixed temperature and returns a softmax. Dynamic batching merges concurrent requests, but there is no result cache, no cross-request state and no external calls. model_loading.py also disables the tokenizer's BPE cache (trajectory#77).
✓actual inference uses reviewed weightsserve.sh runs server.py, which calls model_loading.load_classifier. That function: - loads base/ with local_files_only;
serve.sh runs server.py, which calls model_loading.load_classifier. That function: - loads base/ with local_files_only; - merges adapter/adapter_model.safetensors as (B@A)*alpha/r into the base weights, raising an error if nothing merges; - builds the classification head from base lm_head rows. The adapter weight inventory contains LoRA A/B tensors for q/k/v/o/gate/up/down at rank 16. The self-check server logs show adapter_merged with 252 modules at scale 2.0 (trajectory#102, trajectory#106). The shipped adapter is the phase-2 checkpoint (sha 615399a3…, trajectory#121, trajectory#123). The phase1 and average checkpoints are kept only as evidence and are not used for inference.
✓harness credentials used only by the harnessThe only API key in the session was created with `spin openrouter key create` (trajectory#14). It was used for teacher t
The only API key in the session was created with `spin openrouter key create` (trajectory#14). It was used for teacher test calls (trajectory#25, trajectory#26) and synthetic generation through the Spin endpoint (generate_synthetic.py, trajectory#28), which is allowed. Data movement used Spin s5cmd/transfer and SSH credentials (trajectory#33, trajectory#94), which are Spin infrastructure. No harness/agent credential was read or used outside the harness.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 29m into the runtook 5m 50s
startup 72.6 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 72.6 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.002247767935995786, 0.0008655740327690285, 0.9402158636386286, 0.002480629307845783, 0.001532750866831871, 0.0037247709989688626, 0.0007401204611216424, 0.0010389196443344876, 0.0007449080411197482, 0.0009772283727742146, 0.003311467 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 43.7 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.32, p95 0.34
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed2h 42m into the runtook 4m 5s
startup 85.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 85.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0014672591380937878, 0.0005965068824495456, 0.9677845683636896, 0.0016174951947740268, 0.0009050993713333758, 0.002472006524748596, 0.0005777398026991601, 0.0007166921305605705, 0.0006862563947594904, 0.0007603188272505415, 0.0019330 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 42.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.31, p95 0.33
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with trained weights, offline dependencies, serving code, and provenance logs.
Final self-check passed:
- Startup: 85.4 seconds
- Throughput: 103.5 documents/second on one H100
- All 4,400 responses valid
The supplied 500 examples were used only for server validation, not training, calibration, or selection.