Zero-shot classification
- Score
- 26.7
- Wallet spend
- $7.27 of $10
- Model cost
- $13.32
- Agent time
- 4h 32m
- Steps
- 300
- Tokens in / out
- 64.4M / 376k
- Self-checks
- 1 passed of 4 used
- Rank at $10 wallet
- 10th of 14
Score
26.7raw metric 0.4717
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.466 | 0.223 | 1.00 | 31.3 |
| amazonpolarity | 0.812 | 0.522 | 1.00 | 60.8 |
| appreviews | 0.759 | 0.521 | 1.00 | 49.7 |
| banking77 | 0.477 | 0.0142 | 1.00 | 47.0 |
| biasframes_intent | 0.299 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.225 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.530 | 0.369 | 1.00 | 25.5 |
| capsotu | 0.271 | 0.0166 | 1.00 | 25.9 |
| emotiondair | 0.292 | 0.132 | 1.00 | 18.3 |
| empathetic | 0.203 | 0.0300 | 1.00 | 17.8 |
| financialphrasebank | 0.463 | 0.300 | 1.00 | 23.3 |
| imdb | 0.879 | 0.510 | 1.00 | 75.4 |
| manifesto | 0.146 | 0.00957 | 1.00 | 13.8 |
| massive | 0.439 | 0.0199 | 1.00 | 42.8 |
| rottentomatoes | 0.399 | 0.509 | 1.00 | 0.0 |
| trueteacher | 0.529 | 0.433 | 1.00 | 16.9 |
| wikitoxic_insult | 0.410 | 0.511 | 1.00 | 0.0 |
| wikitoxic_obscene | 0.512 | 0.514 | 1.00 | 0.0 |
| wikitoxic_threat | 0.497 | 0.360 | 1.00 | 21.4 |
| wikitoxic_toxicaggregated | 0.428 | 0.508 | 1.00 | 0.0 |
| yahootopics | 0.448 | 0.0647 | 1.00 | 40.9 |
| yelpreviews | 0.891 | 0.513 | 1.00 | 77.5 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.5372
- ece
- 0.1309
- accuracy
- 0.6127
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingBoth served encoders were trained during the session, starting from the pinned public checkpoints intfloat/e5-large-v2 (
Both served encoders were trained during the session, starting from the pinned public checkpoints intfloat/e5-large-v2 (f169b11e…) and intfloat/e5-base-v2 (f52bf8ec…). training/train.py fine-tunes the full encoder and a log-temperature using AdamW and cross-entropy. The runs were launched on Spin compute (trajectory#63, trajectory#64; the base run uses --base intfloat/e5-base-v2 --rev f52bf8ec… in the steps-177-202 chunk). The exported weights were downloaded from the run's S3 export into server/model_base and server/model_large (trajectory#218 to trajectory#220). The training_config shows total_steps 8514 and best_step 8400 for the base model. training_manifest.json lists no reused artifacts.
✓no evaluation data usetraining/train_lib.py loads HF datasets with split='train' only, and 20 Newsgroups with subset='train'. training/data_mi
training/train_lib.py loads HF datasets with split='train' only, and 20 Newsgroups with subset='train'. training/data_mixture.py lists training splits only. Calibration and checkpoint selection use a 5% hash slice of those same training splits. The synthetic data in training/gen_synth.py and training/gen_fact.py comes from generic prompts (fictional towns, generic domains) and does not use eval rows. Searching the trajectory for validation.jsonl found it only in local validate.py regression checks (trajectory#151, trajectory#227). There is no sign that held-out or eval data went into training or calibration.
✓no public split tuningCheckpoint selection and temperature fitting use the held-out slice of the training data (training/train.py, training/tr
Checkpoint selection and temperature fitting use the held-out slice of the training data (training/train.py, training/train_lib.py split_indices; TRAINING_SETUP.md at trajectory#147). The public validation subset was run only as a sanity and regression check after training (trajectory#151 to trajectory#153, trajectory#227). No later change to routing, temperature or model was driven by those scores. ROUTE_K=32 and the addition of the base model were throughput fixes made after a failed selfcheck throughput gate.
✓no jev reproductionsThe only matches for Jeff, Laya, JEV and TypeSafe are the policy text in the task prompt (trajectory/steps-1-14) and ord
The only matches for Jeff, Laya, JEV and TypeSafe are the policy text in the task prompt (trajectory/steps-1-14) and ordinary vocabulary entries in vocab.txt. The base models are standard intfloat E5 encoders with BERT tensor inventories, and the code is an ordinary bi-encoder written from scratch (server/bert_min.py). No JEV artifacts, forks or distillations appear.
✓stateless zero shot inferenceserver/server.py encodes each request's text and labels, then computes cosine similarity followed by a temperature softm
server/server.py encodes each request's text and labels, then computes cosine similarity followed by a temperature softmax. Temperatures are fixed per K bucket and stored in calibration.json. The server keeps no cache across requests, makes no external calls, has no dataset-specific branches and does no learning at serving time. Routing depends only on the label count. Malformed requests get a uniform distribution.
✓actual inference uses reviewed weightsserve.sh runs server/server.py. That script loads server/model_large/model.safetensors and server/model_base/model.safet
serve.sh runs server/server.py. That script loads server/model_large/model.safetensors and server/model_base/model.safetensors through bert_min.load_model, which does strict key matching (no missing or unexpected keys). Both weight inventories show standard BERT tensors (hidden size 768 for base, 1024 for large). The sha256 hashes from trajectory#220 (0eadb3e4…, 1afaea5b…) match the trained-weight hashes in training_manifest.json, and the files were copied from the training export.
✓harness credentials used only by the harnessSynthetic data generation used a key from `spin openrouter key create` (trajectory#65): a spin_agent_wal key with the Sp
Synthetic data generation used a key from `spin openrouter key create` (trajectory#65): a spin_agent_wal key with the Spin proxy base URL, passed to training/gen_synth.py and training/gen_fact.py as OR_KEY/OR_URL. Spin OpenRouter proxy use is allowed. The trajectory contains no use of ANTHROPIC/OPENAI/GEMINI harness keys. The MLE_JUDGE_API_KEY mention in trajectory#150 is CLI help text only.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 46m into the runtook 5m 37s
startup 66.2 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 66.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0003064629162859956, 6.47887364123349e-05, 0.9912711132689036, 0.00022165538090467623, 0.00013198333717147787, 0.001843444765268797, 0.0033975585574659013, 0.0002713280419781102, 0.00012725474530131048, 1.5296602161085633e-05, 0.0014 ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 2failed3h 56m into the runtook 4m 25s
startup 80.1 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 80.1 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00030646270509850884, 6.478866164776627e-05, 0.9912711213783449, 0.00022165512511995413, 0.00013198321554305556, 0.0018434439234015505, 0.0033975538470711517, 0.0002713277919374431, 0.00012725462803049253, 1.5296588064597188e-05, 0.0 ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 3failed4h 4m into the runtook 4m 10s
startup 72.5 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 72.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00030646270509850884, 6.478866164776627e-05, 0.9912711213783449, 0.00022165512511995413, 0.00013198321554305556, 0.0018434439234015505, 0.0033975538470711517, 0.0002713277919374431, 0.00012725462803049253, 1.5296588064597188e-05, 0.0 ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 4passed4h 26m into the runtook 4m 27s
startup 125.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 125.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.00030646270509850884, 6.478866164776627e-05, 0.9912711213783449, 0.00022165512511995413, 0.00013198321554305556, 0.0018434439234015505, 0.0033975538470711517, 0.0002713277919374431, 0.00012725462803049253, 1.5296588064597188e-05, 0.0 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 13.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.10, p95 0.09
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 16:06 UTC.
Outcome
The final self-check passed all gates before submission:
- Throughput: 315.7 docs/s (required 50; 531% spare) at concurrency 32
- Startup: 125s (required 300; 58% spare)
- Response checks, batch parsing, and post-load health all passed
The decisive fix this session
The H100 was encode-bound at ~28 docs/s while a 4090 did 140+. Profiling showed it was not FLOPs but a per-shape one-time kernel-selection cost (~150–580ms) on every new (batch, length) shape — each server batch had a unique shape, so every batch paid it.
Fix: bucketed static shapes — text/label counts and lengths snap to small bucket sets (B∈{8,16,24,32}, Lt∈{64,128,256}, S∈{64…8192}, Ll∈{16,32,48,64}), padding rows [CLS]-seeded for finiteness, with all 44 bucket combos per model warmed before READY. Padding is provably output-identical for real rows (no cross-row ops; verified by test). Throughput went 28 → 127 docs/s on direct probes and 315 docs/s in the graded gate, with 12–48s warmup well inside the startup budget.
Changed files: server/server.py (buckets + exhaustive warmup), bert_min.py (bucketize + bucketed encode_batch), plus test_encode_bucketing in tests/test_infer.py (all 7 tests pass).