Zero-shot classification
- Score
- 41.2
- Wallet spend
- $13.33 of $200
- Model cost
- $0.96
- Agent time
- 2h 13m
- Steps
- 489
- Tokens in / out
- 46.2M / 313k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 8th of 19
Score
41.2raw metric 0.5875
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.753 | 0.223 | 1.00 | 68.2 |
| amazonpolarity | 0.909 | 0.522 | 1.00 | 81.0 |
| appreviews | 0.808 | 0.521 | 1.00 | 59.8 |
| banking77 | 0.513 | 0.0142 | 1.00 | 50.6 |
| biasframes_intent | 0.449 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.450 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.544 | 0.369 | 1.00 | 27.7 |
| capsotu | 0.469 | 0.0166 | 1.00 | 46.0 |
| emotiondair | 0.495 | 0.132 | 1.00 | 41.8 |
| empathetic | 0.263 | 0.0300 | 1.00 | 24.0 |
| financialphrasebank | 0.500 | 0.300 | 1.00 | 28.5 |
| imdb | 0.807 | 0.510 | 1.00 | 60.7 |
| manifesto | 0.211 | 0.00957 | 1.00 | 20.3 |
| massive | 0.562 | 0.0199 | 1.00 | 55.3 |
| rottentomatoes | 0.743 | 0.509 | 1.00 | 47.6 |
| trueteacher | 0.359 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.742 | 0.511 | 1.00 | 47.2 |
| wikitoxic_obscene | 0.382 | 0.514 | 1.00 | 0.0 |
| wikitoxic_threat | 0.692 | 0.360 | 1.00 | 51.9 |
| wikitoxic_toxicaggregated | 0.808 | 0.508 | 1.00 | 61.0 |
| yahootopics | 0.549 | 0.0647 | 1.00 | 51.7 |
| yelpreviews | 0.917 | 0.513 | 1.00 | 82.8 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6495
- ece
- 0.1039
- accuracy
- 0.6973
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe weights were trained during the session. train_bienc2.py fully fine-tunes Qwen/Qwen3-Embedding-0.6B at the revision
The weights were trained during the session. train_bienc2.py fully fine-tunes Qwen/Qwen3-Embedding-0.6B at the revision pinned in architecture.json, using AdamW over all model parameters plus a learned log_scale. The runs were chained r1 (2500 steps), r2 (2400) and r3 (1500, via --init-from), each with retained logs and state files. training_manifest.json records the shipped model/model.safetensors sha256. The weight inventory shows a BF16 Qwen3-0.6B tensor set that matches model/config.json. finalize_model.py only copies the checkpoint and rewrites the tokenizer config key.
✓no evaluation data useAll training data comes from declared sources, each with a sha256 in training_manifest.json and PROVENANCE.md. fetch_par
All training data comes from declared sources, each with a sha256 in training_manifest.json and PROVENANCE.md. fetch_parquet.py downloads only train splits of 25 public HF datasets, none of them BTZSC sources. The rest is LLM-generated synthetic tasks whose prompts (gen_tasks.py) forbid copying benchmark labels or examples. build_train2.py uses only those train splits and synthetic tasks marked _split=='train'. PROVENANCE.md says BTZSC sources and their derivatives were never downloaded and that DBpedia validation/test splits were excluded. No evaluation data appears in the training builders.
✓no public split tuningNo tuning used public validation or test splits. Checkpoint selection used dev2, built from train-split rows excluded fr
No tuning used public validation or test splits. Checkpoint selection used dev2, built from train-split rows excluded from training by sha1. The temperature (T=0.024) was fitted with fit_calib.py on calib: the 3% train-source holdout plus held-out synthetic tasks, per make_calib.py, which states that nothing comes from a validation or test split. In the trajectory, validation.jsonl was read only for length/throughput sizing (step 408) and to look at the label format of 2 rows after the r3/r4 comparison was already done (step 473). The keep-r3 decision rested on dev2/calib scores (steps 471-474). The validation set was otherwise used only by the validate.py self-check.
✓no jev reproductionsThe manifest's reused_artifacts list is empty. The base is the ordinary pretrained encoder Qwen3-Embedding-0.6B, and the
The manifest's reused_artifacts list is empty. The base is the ordinary pretrained encoder Qwen3-Embedding-0.6B, and the training and serving code (a bi-encoder contrastive fine-tune with last-token pooling) is independently written. No Jeff, Laya, Open-Jev, JEV-4B or typesafe-diffusion-lab artifacts are loaded or referenced in the code. The only trajectory matches for those names are the policy text itself.
✓stateless zero shot inferenceEach request is scored independently in server.py. The text and that request's label descriptions are encoded fresh, and
Each request is scored independently in server.py. The text and that request's label descriptions are encoded fresh, and probabilities come from a softmax over cosine similarity divided by a fixed temperature. Label embeddings are never cached across requests, and there is no retrieval, adaptation, state mutation or external call. Dynamic batching only groups independent requests for throughput.
✓actual inference uses reviewed weightsserve.sh installs the vendored, audited wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, and runs exec python3
serve.sh installs the vendored, audited wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, and runs exec python3 server.py. server.py loads AutoModel/AutoTokenizer from ./model with local_files_only. The reviewed model/model.safetensors inventory (Qwen3-0.6B BF16 tensors) matches model/config.json and the manifest sha256, and the trajectory confirms the sha matched before the validate run. All 40 wheels carry approved trusted binary audits.
✓harness credentials used only by the harnessNo harness credentials were used by the participant. Teacher-LLM calls in gen_tasks.py used a participant key created wi
No harness credentials were used by the participant. Teacher-LLM calls in gen_tasks.py used a participant key created with `spin openrouter key create` (trajectory step 111) against the Spin proxy, and fetch_parquet.py uses HF_TOKEN only for dataset downloads. Trajectory searches for ANTHROPIC, BASE_URL, OPENAI_API_KEY and `claude -p` found no use of the agent harness's own credentials, and the participant's secret scan (step 411) found nothing beyond HF_TOKEN.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 30m into the runtook 3m 15s
startup 64.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 64.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.011207657242248831, 0.0017917223582355013, 0.6492268734287432, 0.003262765019990515, 0.05579628393645742, 0.032162468142614, 0.04045419578657004, 0.0028967981397892135, 0.002417588038164547, 0.0019291168661430356, 0.0322396710480461, ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 15.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.11, p95 0.12
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed2h 6m into the runtook 3m 5s
startup 80.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 80.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.011207657242248831, 0.0017917223582355013, 0.6492268734287432, 0.003262765019990515, 0.05579628393645742, 0.032162468142614, 0.04045419578657004, 0.0028967981397892135, 0.002417588038164547, 0.0019291168661430356, 0.0322396710480461, ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 17.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.12, p95 0.13
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output (87 files, 5.16 GB) was graded as-is.
What shipped
serve.sh+server.py— offline ThreadingHTTPServer with a dynamic batching worker (≤64 requests, 4 ms wait), bf16 SDPA inference, per-request label embedding (no cross-request caching),TEMP=0.024, last-token pooling with the Qwen3 query instruction.model/— r3 checkpoint (Qwen3-Embedding-0.6B fully fine-tuned, sha256cc61bdf4…), plus tokenizer/config patches for transformers 4.57.1.wheels/— 40 unmodified cp312 PyPI wheels installed into/tmpat startup (--no-index).training_manifest.json,architecture.json,training/(code, config, PROVENANCE with per-corpus parquet hashes and exclusions, r1–r3 logs/states,raw_sha256.txt),evidence/(throughput bench, validation run, temperature fits, model-selection tables, sims).
Final verification
frontier selfcheck(id45ae4b1f83a8): passed — startup 80.2 s of 300 s; 4400/4400 responses parsed; throughput 257.3 docs/s vs 50 required (4.1× margin); digest3cc27e21….- Earlier worst-case mixed benchmark (1642-char texts, 66 labels/request mean): 77.5 docs/s, 0 errors.
- Local
/task/validate.pyon DBpedia-14: 500/500 requests, macro-F1 0.968, ECE 0.047, 270.8 docs/s.
Training selection
- r3 (1,500 steps from r2; 25 real train-split corpora + 644 synthetic teacher tasks) beat r1/r2 on both held-out suites (dev2 0.6421, calib 0.5755 at T=0.025).
- An extra r4 round on ~800 newly generated synthetic tasks was run and rejected: +0.006 on the synthetic-heavy calib suite but −0.004 on the real-source dev2 suite, whose templated descriptions match the public validation format. The shipped weights remain r3; the experiment is documented in
training/PROVENANCE.md.