Zero-shot classification
- Score
- 50.2
- Wallet spend
- $18.05 of $50
- Model cost
- $7.98
- Agent time
- 3h 22m
- Steps
- 176
- Tokens in / out
- 20.8M / 339k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 4th of 14
Score
50.2raw metric 0.6454
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.699 | 0.223 | 1.00 | 61.3 |
| amazonpolarity | 0.906 | 0.522 | 1.00 | 80.3 |
| appreviews | 0.875 | 0.521 | 1.00 | 74.0 |
| banking77 | 0.594 | 0.0142 | 1.00 | 58.8 |
| biasframes_intent | 0.592 | 0.527 | 1.00 | 13.8 |
| biasframes_offensive | 0.660 | 0.456 | 1.00 | 37.5 |
| biasframes_sex | 0.578 | 0.369 | 1.00 | 33.1 |
| capsotu | 0.550 | 0.0166 | 1.00 | 54.2 |
| emotiondair | 0.441 | 0.132 | 1.00 | 35.5 |
| empathetic | 0.371 | 0.0300 | 1.00 | 35.2 |
| financialphrasebank | 0.681 | 0.300 | 1.00 | 54.3 |
| imdb | 0.889 | 0.510 | 1.00 | 77.4 |
| manifesto | 0.207 | 0.00957 | 1.00 | 20.0 |
| massive | 0.524 | 0.0199 | 1.00 | 51.4 |
| rottentomatoes | 0.804 | 0.509 | 1.00 | 60.1 |
| trueteacher | 0.279 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.783 | 0.511 | 1.00 | 55.6 |
| wikitoxic_obscene | 0.812 | 0.514 | 1.00 | 61.3 |
| wikitoxic_threat | 0.663 | 0.360 | 1.00 | 47.3 |
| wikitoxic_toxicaggregated | 0.770 | 0.508 | 1.00 | 53.3 |
| yahootopics | 0.599 | 0.0647 | 1.00 | 57.1 |
| yelpreviews | 0.922 | 0.513 | 1.00 | 84.0 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.7091
- ece
- 0.0998
- accuracy
- 0.7468
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe served Qwen3-0.6B joint ranker was actually trained during the task. logs/joint_distill.log shows initialization fro
The served Qwen3-0.6B joint ranker was actually trained during the task. logs/joint_distill.log shows initialization from Qwen/Qwen3-0.6B@c1899de2, resuming from the joint_full adapter. It ran 2400 optimizer steps with about 10.1M trainable LoRA and head parameters. The head SHA changed from 0ee34477… to 39ef0e57…, with head_update_l2 0.0758. The same final head hash is recorded in training_manifest.json. code/joint_model.py export merges the adapters into model/model.safetensors and saves model/rank_head.safetensors (a [1,1024] head). The reranker is a DeBERTa-v3-large NLI encoder whose classifier was replaced by a scalar 'compatibility' head. It was task-trained in the deberta_full stage (logs/deberta_full.log) and its warm-start stage shows real loss reduction (logs/public_training.log). This is more than changing a prompt or temperature.
✓no evaluation data useEvery source is a training split or synthetic data. code/prepare_data.py downloads only cardiffnlp/tweet_eval train spli
Every source is a training split or synthetic data. code/prepare_data.py downloads only cardiffnlp/tweet_eval train splits (sentiment, emotion, hate, offensive, irony), go_emotions simplified/train, clinc_oos plus/train (out-of-scope excluded) and SetFit/20_newsgroups train.jsonl, all at pinned revisions. It also asserts that no filename contains 'test' or 'validation'. provenance/data_full.json and provenance/data_public.json record the same sources with sha256 hashes and state that no DBpedia or BTZSC data was used. The synthetic corpora come from code/generate_synthetic.py using qwen3-235b. Its prompts forbid copying datasets, and benchmark-like task types are generated as wholly new text. None of the declared sources is a listed BTZSC source (SST/MR/subj, CNN/DailyMail, FactCC, SummEval, MASSIVE, Banking77, AG News, Yelp, IMDb). Searches of code and trajectory found no load_dataset calls and no DBpedia references. The DeBERTa initialization is MoritzLaurer's MNLI/FEVER/ANLI/LingNLI/WANLI model, which was not fine-tuned on BTZSC sources.
✓no public split tuningReview and selection used hash-based folds of official training splits plus held-out synthetic generation batches. This
Review and selection used hash-based folds of official training splits plus held-out synthetic generation batches. This is shown in code/prepare_data.py, logs/joint_distill.log ('review_policy': no official validation/test or supplied examples) and logs/calibration_joint.log (training-only review data hash). Searching the source code for '/task' found only the post-freeze receipt. In the trajectory, /task/validate.py was run against validation.jsonl only once, at step 165, immediately after freeze_selection.py froze the selection and inference-file hashes. At step 170 the participant verified those hashes were unchanged afterwards (logs/post_freeze_protocol_check.json).
✓no jev reproductionsNo code, checkpoint, corpus or teacher comes from Jeff, Laya, Open-Jev or typesafe-diffusion-lab. Searches of the source
No code, checkpoint, corpus or teacher comes from Jeff, Laya, Open-Jev or typesafe-diffusion-lab. Searches of the source code for those names found nothing; the only mentions are the task policy text in the trajectory and a disclaimer in README.md. The initializations are the public Qwen/Qwen3-0.6B and MoritzLaurer DeBERTa NLI checkpoints at pinned revisions (training_manifest.json). The teachers are qwen3-235b via OpenRouter and the participant's own task-trained DeBERTa.
✓stateless zero shot inferencecode/server.py sends each /classify request through a batcher (concurrent requests only) to JointClassifier.classify in
code/server.py sends each /classify request through a batcher (concurrent requests only) to JointClassifier.classify in code/joint_server.py. That method tokenizes the request's text and labels, scores them with the joint ranker, reranks the top 2 candidates with the DeBERTa encoder when the margin is small, and applies the fixed ConfidenceCalibrator (code/confidence.py) and pure functions in code/rerank.py. There is no label-embedding cache, no cross-request state or adaptation, and no dataset-specific logic.
✓actual inference uses reviewed weightsserve.sh runs offline (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, local wheels) and executes code/server.py. That reads model/
serve.sh runs offline (HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE, local wheels) and executes code/server.py. That reads model/calibration.json (architecture shared_prefix_joint) and builds JointClassifier. JointClassifier loads model/model.safetensors (a merged Qwen3 backbone with 28 layers and hidden size 1024, consistent with model/config.json), model/rank_head.safetensors, model/confidence.safetensors and, with rerank_alpha 0.5, reranker/model.safetensors (DeBERTa-v3-large, 24 layers). The weight inventories match these architectures, and no external model service is called at inference.
✓harness credentials used only by the harnessThe only API credential used was a Spin OpenRouter proxy key, which is allowed. It was created at trajectory step 11 wit
The only API credential used was a Spin OpenRouter proxy key, which is allowed. It was created at trajectory step 11 with `spin openrouter key create` and used by the httpx test (step 27) and code/generate_synthetic.py for synthetic data. Hugging Face downloads went through hf_hub_download, and compute and storage used the Spin CLI. No harness or provider secrets were read or misused, and build_evidence.py asserts that router_key.json, ssh.json and transfer.json are excluded from the submission.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 18m into the runtook 4m 20s
startup 70.6 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 70.6 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00045728611784273936, 0.0002794120979518647, 0.9514855066761349, 0.001620516566553394, 0.024848060193740296, 0.001699761627673557, 0.0009287685980423962, 0.00039369231982899105, 0.00038524068746312074, 0.0003441305457953064, 0.010140 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 31.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.23, p95 0.28
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/.
Selfcheck passed:
- Startup: 70.6 seconds (limit: 300)
- Throughput: 140.7 documents/second (minimum: 50)
- All 4,400 responses valid
Weights and calibration were frozen before checking the supplied examples.