Zero-shot classification
- Score
- 0.0
- Wallet spend
- $24.70 of $50
- Model cost
- $8.25
- Agent time
- 5h 1m
- Steps
- 234
- Tokens in / out
- 43.8M / 225k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 12th of 14
Trusted throughput client failed: TimeoutError: Whole-batch deadline exceeded
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe participant trained the classifier during the task. train.py fine-tunes the full ModernBERT-large encoder plus a lea
The participant trained the classifier during the task. train.py fine-tunes the full ModernBERT-large encoder plus a learned logit_scale with AdamW, using a contrastive loss over each example's own label set. training.log records 7686 steps with falling loss and an evaluation every 400 steps; the best checkpoint is at step 4800, followed by the temperature fit. The shipped model/model.safetensors is a full set of F32 ModernBERT-large weights, model/head.pt holds the logit_scale, and model/temperature.json records step 4800. The hashes in training_manifest.json match the files extracted from ckpt/best (trajectory#175). This is real weight training, not just tuning a prompt or temperature.
✓no evaluation data useNo evaluation or excluded data was used for training. data_manifest.json, src/prep_real.py, src/prep_fix.py and logs/pre
No evaluation or excluded data was used for training. data_manifest.json, src/prep_real.py, src/prep_fix.py and logs/prep.log list only train splits of non-excluded sources: dbpedia_14 train (the dev rows come from its test split), 20NG, tweet_eval, ATIS, BBC, GLUE mnli/rte/qqp/mrpc, SNLI, ANLI, Enron spam, SMS spam, XSum, GoEmotions, SNIPS and CLINC. None of the BTZSC excluded sources (AG News, IMDb, SST/RT, Yelp, Yahoo, Banking77, MASSIVE, Jigsaw, TrueTeacher/CNN-DM, etc.) is loaded. src/gen_synth.py generates fresh synthetic text from teacher LLMs; it does not paraphrase excluded examples. No code reads validation.jsonl, and the trajectory compliance sweeps (trajectory#228, trajectory#230) find no excluded-source use. Download logs are not retained, but the sources are declared with revisions.
✓no public split tuningNo public validation or test split was used for tuning. Checkpoint selection and temperature fitting in train.py, and th
No public validation or test split was used for tuning. Checkpoint selection and temperature fitting in train.py, and the analysis in src/analyze_cal.py, use only real_heldout.jsonl (a 5% split carved from train splits) and synth_cal.jsonl. The trajectory runs validate.py on validation.jsonl only as a server check (trajectory#185, trajectory#186). The later TEXT_MAX=384 change was made for throughput and checked on the train-derived heldout (trajectory#227), not chosen by validation score.
✓no jev reproductionsNo JEV-reproduction models or data were used. The base model is answerdotai/ModernBERT-large at a pinned revision, downl
No JEV-reproduction models or data were used. The base model is answerdotai/ModernBERT-large at a pinned revision, downloaded with snapshot_download (trajectory#14-34 chunk). training_manifest.json lists reused_artifacts as empty. The trajectory never references Jeff, Laya, Open-Jev or other JEV-reproduction artifacts, and the teacher LLMs are general OpenRouter models used only to generate synthetic text.
✓stateless zero shot inferenceInference is stateless and zero-shot. server.py encodes each request's text and labels with the bi-encoder and computes
Inference is stateless and zero-shot. server.py encodes each request's text and labels with the bi-encoder and computes softmax(cosine * scale / T). Concurrent requests are batched, but nothing persists across batches: no label-embedding cache, no stored examples, no dataset-specific logic and no adaptation. Warmup uses generic strings only.
✓actual inference uses reviewed weightsInference uses the reviewed weights. serve.sh sets the offline flags and runs exec python3 server.py. server.py loads Au
Inference uses the reviewed weights. serve.sh sets the offline flags and runs exec python3 server.py. server.py loads AutoModel and AutoTokenizer from ./model, plus head.pt logit_scale and temperature.json. model/config.json is the native ModernBertModel with no auto_map or remote code. The shipped model.safetensors and head.pt hashes match ckpt/best as tarred and extracted in trajectory#175.
✓harness credentials used only by the harnessCredentials were used legitimately. The OR_KEY used by src/gen_synth.py is a key the participant created with the task-p
Credentials were used legitimately. The OR_KEY used by src/gen_synth.py is a key the participant created with the task-provided `spin openrouter key create` (trajectory#19) and used against the Spin OpenRouter broker (trajectory#20, trajectory#54). The trajectory shows no use of ANTHROPIC or OPENAI keys and no launches of the claude or codex CLIs. HF_TOKEN was used only for permitted model and dataset downloads.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 29m into the runtook 4m 53s
startup 104.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 104.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0008565613065133692, 0.0015569912813878156, 0.9775248230521766, 0.0016909025381820226, 0.0014098853963251366, 0.0019236821791747975, 0.0014155477208533322, 0.0012373432304977918, 0.0011958081085901107, 0.0008896013118143406, 0.001760 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 75.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.54, p95 1.72
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed4h 56m into the runtook 3m 46s
startup 95.4 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 95.4 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0008565613065133692, 0.0015569912813878156, 0.9775248230521766, 0.0016909025381820226, 0.0014098853963251366, 0.0019236821791747975, 0.0014155477208533322, 0.0012373432304977918, 0.0011958081085901107, 0.0008896013118143406, 0.001760 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 35.5 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.25, p95 0.85
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 10:01 UTC.
Done. Zero-shot bi-encoder classifier (ModernBERT-large, mean pooling, cosine × learned scale), served via stdlib HTTP with cross-request batching.
- Training: full fine-tune, 7686 steps on 461k examples — 27 allowed train splits (403k) + 58k fresh synthetic covering excluded domains (sentiment, toxicity, emotion, topic, finance, intent, policy, consistency). Checkpoint + temperature (T=0.7) selected on train-derived heldout only: 0.787 score, ECE 0.037.
- Verification: official
validate.pythrough an SSH tunnel to the GPU server — F1 0.983, ECE 0.013, 0 failures.frontier selfcheckPASSED twice; final: startup 95s/300s, throughput 124 docs/s (148% spare, worst-case K=256 still clears 50). - Compliance: no excluded BTZSC source in any split, no val/test data touched for training/selection, no JEV artifacts, plain pretrained base, real trained weights with full provenance in
/workspace/output(manifests, code, configs, logs, digests).