Zero-shot classification
- Score
- 41.1
- Wallet spend
- $28.48 of $50
- Model cost
- $29.18
- Agent time
- 5h 16m
- Steps
- 429
- Tokens in / out
- 41.8M / 273k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 7th of 14
Score
41.1raw metric 0.5783
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.689 | 0.223 | 1.00 | 59.9 |
| amazonpolarity | 0.905 | 0.522 | 1.00 | 80.1 |
| appreviews | 0.814 | 0.521 | 1.00 | 61.2 |
| banking77 | 0.525 | 0.0142 | 1.00 | 51.8 |
| biasframes_intent | 0.437 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.542 | 0.456 | 1.00 | 15.8 |
| biasframes_sex | 0.437 | 0.369 | 1.00 | 10.8 |
| capsotu | 0.494 | 0.0166 | 1.00 | 48.5 |
| emotiondair | 0.423 | 0.132 | 1.00 | 33.5 |
| empathetic | 0.243 | 0.0300 | 1.00 | 21.9 |
| financialphrasebank | 0.408 | 0.300 | 1.00 | 15.4 |
| imdb | 0.849 | 0.510 | 1.00 | 69.2 |
| manifesto | 0.139 | 0.00957 | 1.00 | 13.0 |
| massive | 0.526 | 0.0199 | 1.00 | 51.6 |
| rottentomatoes | 0.790 | 0.509 | 1.00 | 57.2 |
| trueteacher | 0.283 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.741 | 0.511 | 1.00 | 47.0 |
| wikitoxic_obscene | 0.858 | 0.514 | 1.00 | 70.8 |
| wikitoxic_threat | 0.432 | 0.360 | 1.00 | 11.3 |
| wikitoxic_toxicaggregated | 0.776 | 0.508 | 1.00 | 54.5 |
| yahootopics | 0.481 | 0.0647 | 1.00 | 44.5 |
| yelpreviews | 0.930 | 0.513 | 1.00 | 85.7 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6559
- ece
- 0.1354
- accuracy
- 0.7007
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe shipped cross-encoder was trained during the task. Training started from the public MoritzLaurer/DeBERTa-v3-base-mnl
The shipped cross-encoder was trained during the task. Training started from the public MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli NLI checkpoint, as recorded in cfg_run1 init_model and train/train.py, which uses from_pretrained with num_labels=1 and copies the head from the entailment row. Training then continued in stages: run1 → run2 → run3 → run5/6/7, with each stage's init_model pointing at the previous best. The trajectory shows the full sequence: the runs being created and executed, the 4-member weight soup being built from run3/5/6/7 best (trajectory#412), and the soup being packaged into model/xenc_a with sha256 283e7485…, which matches training_manifest.json (trajectory#417). The weight inventory of model/xenc_a/model.safetensors confirms a DeBERTa-v2 encoder with a 1×768 classifier, matching that design. An earlier attempt to download the ling-wanli variant returned 404 and was never used. model/bge is the declared unmodified BAAI/bge-base-en-v1.5 encoder, which serves only as a prefilter.
✓no evaluation data useAll training corpora are declared in training_manifest.json and train/data_manifest.json, and train/prep_real.py and tra
All training corpora are declared in training_manifest.json and train/data_manifest.json, and train/prep_real.py and train/prep_more.py load only their train splits. The corpora are 20NG, DBpedia-14, TREC, CLINC150, GoEmotions, TweetEval, Davidson, CFPB, SNIPS, MNLI, FEVER-NLI, SNLI, VitaminC and twitter-financial-news-topic. None of them is a BTZSC source dataset on the forbidden list. The synthetic data comes from llama-4-scout through the Spin proxy using a generic task-generation prompt (train/gen_synth.py). I found no use of validation or test splits: no reads with split=test.
✓no public split tuning/task/public/validation.jsonl was used only as a smoke check, run through validate.py against local servers (trajectory#
/task/public/validation.jsonl was used only as a smoke check, run through validate.py against local servers (trajectory#56-57, plus later validate runs). Temperature and alpha were selected by train/calib_ckpt.sh and train/score_server.py on 10% md5 dev holdouts of the train splits (data/*.dev.jsonl). train/calib_soup4.log records the chosen values, T=0.65 and alpha=0.02.
✓no jev reproductionstraining_manifest.json lists reused_artifacts as []. The only checkpoints downloaded were MoritzLaurer NLI models, BAAI/
training_manifest.json lists reused_artifacts as []. The only checkpoints downloaded were MoritzLaurer NLI models, BAAI/bge-base-en-v1.5 and an exploratory ModernBERT-base/large, which was not used. 'jeff' and 'laya' appear in the trajectory only inside the quoted policy text, and no JEV, Jeff, Laya or typesafe artifacts were fetched.
✓stateless zero shot inferencePer serve_config.json, server.py scores each request independently. It tokenizes [CLS] text [SEP] label [SEP] pairs, run
Per serve_config.json, server.py scores each request independently. It tokenizes [CLS] text [SEP] label [SEP] pairs, runs the cross-encoder, takes a per-request softmax(logits/T), applies the uniform alpha blend and normalizes. A bge top-48 prefilter applies only when a request has more than 160 labels. The server keeps no cache across requests, does no dataset identification, makes no external calls and does no test-time learning. Warmup uses generic labels.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE=1 and execs server.py with serve_config.json. That co
serve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE=1 and execs server.py with serve_config.json. That config points cross_encoders at model/xenc_a and bi_encoder at model/bge, and server.py loads both with from_pretrained from those local paths. Weight inventories exist for model/xenc_a/model.safetensors (DeBERTa-v2 encoder, fp16, 1-logit classifier) and model/bge/model.safetensors. The serving path contains no remote model calls.
✓harness credentials used only by the harnessThe only API key used was created by the participant with `spin openrouter key create` and used against the Spin proxy i
The only API key used was created by the participant with `spin openrouter key create` and used against the Spin proxy in train/gen_synth.py (trajectory#94). S3 transfer credentials came from `spin transfer credentials`. HF_TOKEN was used only for HF downloads, which is allowed. Searches found no use of OPENAI_API_KEY or Anthropic harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 47m into the runtook 3m 49sfree (stopped before any GPU time)
failed training evidence: ValueError: Evidence path missing, symlinked or outside submission
2 gates
- ✓review prerequisites
- ✗training evidence · ValueError: Evidence path missing, symlinked or outside submission
- 2passed2h 51m into the runtook 3m 9s
startup 87.1 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 87.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0040068987107056, 0.00430630841693117, 0.9434422726449686, 0.004348772969020309, 0.00390812469219295, 0.004776599007420916, 0.004202828765834218, 0.004320394054844868, 0.0038661096429012257, 0.0038661096429012257, 0.00479298692108380 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 28.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.21, p95 0.26
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 3passed3h 28m into the runtook 3m 21s
startup 99.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 99.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.002510197061495188, 0.0025532841841102456, 0.965489920902665, 0.002792064849241982, 0.0025011812223562123, 0.0028622922420036994, 0.0024981922570839984, 0.0025851073832820048, 0.0023641183573834936, 0.0023307917197462496, 0.002830644 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 28.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 0.23
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 4passed4h 44m into the runtook 3m 10s
startup 69.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 69.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0019097952632301934, 0.002018888135193776, 0.9738168546579045, 0.0021614686313345703, 0.0019215032737360414, 0.0020902911582950873, 0.0019457808800117386, 0.002010085693213294, 0.0018565807933444306, 0.0018126978440490282, 0.00213127 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 22.8 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.17, p95 0.19
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 5passed5h 14m into the runtook 3m 9s
startup 71.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 71.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.001881925617953866, 0.001993118965626227, 0.9742423100631061, 0.002135821922446225, 0.001897160713358081, 0.002040829930214181, 0.0019306903758458476, 0.001976407783804621, 0.001831792985185227, 0.0017713986811010198, 0.0020965750740 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 22.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.16, p95 0.19
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 13:38:41 UTC — session complete.
Final summary
Shipped artifact: a zero-shot cross-encoder classifier, where the final weights are a model soup (equal-quarters weight average) of four same-lineage checkpoints — a strong parent plus three siblings trained with different seeds on an expanded data mix.
How the last stretch went:
- Prepared three new allowed datasets (SNLI, VitaminC, twitter-financial-news-topic — none BTZSC sources, train splits only) and ran run5 from run3/best: 0.7071 official-dev, slightly below run3's 0.7096
- run4 (same-data polish) was detected as overfitting and killed early
- Souping was the win: run3+run5 (0.7115) → +run6 (0.7123) → +run7 (0.7129, macro-F1 0.7826 — best on both metrics). Control experiments (children-only soup,
last-checkpoint soup) confirmed the parent-anchored best-checkpoint recipe - Calibration re-tuned through the real HTTP path on the exact official metric: T=0.65, α=0.02 (sweep grid extended below 0.7 since run3's optimum sat on the edge)
Final gates (last selfcheck, all passed):
- Throughput: 194.5 docs/s (required 50 — 289% margin)
- Startup: 71.3s (allowed 300s)
- All 4400 batch responses parsed; healthy after load
- Training evidence + prerequisites: pass
Provenance: 16 training corpora with sha256 digests (15 public train splits + synthetic llama-4-scout tasks), full stage-by-stage configs/logs, soup merge documented in training_manifest.json; no BTZSC source dataset, no validation/test split of any public dataset used for training, calibration, or selection.