Zero-shot classification
- Score
- 45.2
- Wallet spend
- $1.80 of $10
- Model cost
- $0.66
- Agent time
- 2h 29m
- Steps
- 317
- Tokens in / out
- 30.0M / 262k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 8th of 19
Score
45.2raw metric 0.6086
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.735 | 0.223 | 1.00 | 65.9 |
| amazonpolarity | 0.931 | 0.522 | 1.00 | 85.5 |
| appreviews | 0.869 | 0.521 | 1.00 | 72.7 |
| banking77 | 0.415 | 0.0142 | 1.00 | 40.7 |
| biasframes_intent | 0.468 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.573 | 0.456 | 1.00 | 21.5 |
| biasframes_sex | 0.536 | 0.369 | 1.00 | 26.4 |
| capsotu | 0.522 | 0.0166 | 1.00 | 51.3 |
| emotiondair | 0.417 | 0.132 | 1.00 | 32.8 |
| empathetic | 0.333 | 0.0300 | 1.00 | 31.3 |
| financialphrasebank | 0.566 | 0.300 | 1.00 | 38.0 |
| imdb | 0.915 | 0.510 | 1.00 | 82.7 |
| manifesto | 0.133 | 0.00957 | 1.00 | 12.5 |
| massive | 0.527 | 0.0199 | 1.00 | 51.7 |
| rottentomatoes | 0.785 | 0.509 | 1.00 | 56.3 |
| trueteacher | 0.313 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.758 | 0.511 | 1.00 | 50.4 |
| wikitoxic_obscene | 0.800 | 0.514 | 1.00 | 58.9 |
| wikitoxic_threat | 0.554 | 0.360 | 1.00 | 30.3 |
| wikitoxic_toxicaggregated | 0.751 | 0.508 | 1.00 | 49.5 |
| yahootopics | 0.573 | 0.0647 | 1.00 | 54.3 |
| yelpreviews | 0.914 | 0.513 | 1.00 | 82.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6678
- ece
- 0.0990
- accuracy
- 0.7164
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingTwo training phases ran during the task: - Phase 1: training/train.py fully fine-tuned the encoder and head (AdamW, cosi
Two training phases ran during the task: - Phase 1: training/train.py fully fine-tuned the encoder and head (AdamW, cosine schedule) from ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli for 3000 steps. - Phase 2: 1200 more steps from the phase-1 final checkpoint. training/training_config.json records 4200 total steps, and training/training.log logs both phases. The trajectory shows phase 1 finishing ('saved final /workspace/runs/rob/final') and phase 2 starting (trajectory#195). It also shows the rob2/final checkpoint being uploaded and downloaded with sha256 7fccaf94…, which matches training_manifest.json and the shipped model/model.safetensors (trajectory#296, trajectory#298). The weight inventory is a full RoBERTa-large model with a 3-way classifier head.
✓no evaluation data usedata_manifest.json, training/data.py and training/download_commands.txt declare only train splits at pinned revisions: -
data_manifest.json, training/data.py and training/download_commands.txt declare only train splits at pinned revisions: - Classification: dbpedia_14, 20_newsgroups, bbc-news, TREC-QC, clinc_oos, go_emotions, tweet_eval subsets, poem_sentiment, GLUE mrpc/cola, boolq. - NLI: multi_nli, snli, anli, WANLI. None of these is on the BTZSC exclusion list (no AG News, Yahoo, IMDb, SST/MR, Banking77, CNN/DM and so on). training/data.py loads split='train' only and does not read the public validation file. The 'Generating validation split' lines in training/build.log are the datasets library materializing every split while only train is used. The calibration and selection corpora are held-out rows of those same training splits. Download logs are retained in training/build.log and training/download_commands.txt.
✓no public split tuningThe participant viewed /task/public/validation.jsonl rows for format only (trajectory#3, trajectory#102). The public val
The participant viewed /task/public/validation.jsonl rows for format only (trajectory#3, trajectory#102). The public validation rows were scored exactly once, by the local client check at trajectory#290. That run came after: - the phase-2 config was fixed and phase-2 training had started (trajectory#195); - the calibrator T(K) had been fitted on calib_dedup.jsonl (the temperature is already in effect at trajectory#288). No checkpoint, temperature or config was changed after that score. The next steps only upload the model and build the output (trajectory#296-301). Checkpoint and profile comparisons used proxy.jsonl, which is held-out training-split rows (training/eval.py; trajectory#196, trajectory#200). training/calib.py fits on calib.jsonl. Shifting phase 2 toward the uniform 'This text describes {label}' template was informed by seeing the public rows' format, not by validation scores or labels, so it is not established selection or tuning on the public split.
✓no jev reproductionsarchitecture.json and training_manifest.json declare the base as ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli at
architecture.json and training_manifest.json declare the base as ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli at pinned revision 5b605abab9…, an NLI cross-encoder that predates the benchmark and is not a JEV reproduction. teacher_models and reused_artifacts are empty, and the training logs show loading from that base. No reproduction checkpoints appear anywhere in the trajectory.
✓stateless zero shot inferenceserver.py handles each request independently: - It tokenizes the request's text against its own labels and scores them w
server.py handles each request independently: - It tokenizes the request's text against its own labels and scores them with scoring.score_id_pairs. - It applies softmax with T(K), which depends only on the number of labels in that request, plus fixed eps smoothing from calibrator.json. The Worker thread only batches concurrent requests. There is no cache that persists across requests, no retrieval, no dataset identification and no online adaptation. The per-call caches in scoring.score_pairs are local to a single call and are not used for serving. serve.sh runs offline (HF offline env vars, local wheels).
✓actual inference uses reviewed weightsserve.sh execs server.py, which loads ./model through scoring.load_seq_model with local_files_only. model/model.safetens
serve.sh execs server.py, which loads ./model through scoring.load_seq_model with local_files_only. model/model.safetensors is in the weight inventory (1421499516 bytes, a full roberta.* model plus the classifier head). training_manifest.json records sha256 7fccaf948e… for that file, and the trajectory shows the same hash for the downloaded phase-2 final checkpoint (trajectory#298) that build_output copied into the output (trajectory#300-301). Wheel audits (e.g. wheels/annotated_doc-0.0.5-py3-none-any.whl) are approved upstream digests.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENAI and BASE_URL turned up only library source text and the spin CLI help t
Searches of the trajectory for ANTHROPIC, OPENAI and BASE_URL turned up only library source text and the spin CLI help text. The read-only HF_TOKEN was used only to download the declared public models and datasets (trajectory#1, trajectory/steps-14-22). The spin session credentials were used only through the provided spin compute/s5cmd transfer tooling to move the participant's own checkpoint (trajectory#295-298). No credential misuse was found.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 24m into the runtook 2m 14s
startup 28.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 28.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [6.730710728512653e-07, 9.409522334782611e-07, 0.9999264626395707, 1.1291247047704335e-05, 2.431288094679994e-06, 1.4117127725826696e-05, 3.800539863714397e-06, 7.958189048326298e-07, 6.730710728512653e-07, 7.958189048326298e-07, 1.1291 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 23.3 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.17, p95 1.06
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed2h 27m into the runtook 2m 14s
startup 35.6 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 35.6 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [6.730710728512653e-07, 9.409522334782611e-07, 0.9999264626395707, 1.1291247047704335e-05, 2.431288094679994e-06, 1.4117127725826696e-05, 3.800539863714397e-06, 7.958189048326298e-07, 6.730710728512653e-07, 7.958189048326298e-07, 1.1291 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 25.2 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.18, p95 1.07
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state:
Solution — NLI cross-encoder (ynie/roberta-large-snli_mnli_fever_anli_R1_R2_R3-nli, pinned revision) fine-tuned in two phases (3000 + 1200 steps) on training splits of 15 independent classification corpora plus 6 NLI corpora, scoring (text, label description) entailment logits and normalizing over each request's labels with a label-count-dependent temperature T(K) = 1.5 − 0.1·log2 K.
Verified on the grading image (H100, 2 selfchecks):
- startup: 35.6 s (limit 300 s)
- throughput: 174.5 docs/s at concurrency 32 (required 50), 4,400 docs in 25.2 s of 88 s
- all 4,400 responses parsed;
/classifyshape and health checks pass - local server check on the 500 public rows: 67.3 docs/s on the RTX 4090, valid sums, edge cases (2/3/256 labels) pass
Artifacts in /workspace/output/ (75 files, 5.39 GB) — serve.sh, server/scoring code, safetensors weights (sha256 in manifest), calibrator.json, 49 unmodified cp312 wheels with per-file hashes installed to /tmp at startup, training_manifest.json, data_manifest.json, architecture.json, and training code/config/logs plus download commands and the selection/calibration evidence.
Provenance: only training splits of public corpora not on the exclusion list; the 500 public validation rows were used solely for the local server check, never for training, calibration, or model selection. Submitted at 2026-10-20 22:06:49 UTC.