Zero-shot classification
- Score
- 37.1
- Wallet spend
- $14.85 of $200
- Model cost
- $12.51
- Agent time
- 5h 56m
- Steps
- 180
- Tokens in / out
- 27.5M / 146k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 8th of 14
Score
37.1raw metric 0.5539
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.630 | 0.223 | 1.00 | 52.4 |
| amazonpolarity | 0.863 | 0.522 | 1.00 | 71.3 |
| appreviews | 0.842 | 0.521 | 1.00 | 67.0 |
| banking77 | 0.400 | 0.0142 | 1.00 | 39.1 |
| biasframes_intent | 0.458 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.548 | 0.456 | 1.00 | 17.0 |
| biasframes_sex | 0.301 | 0.369 | 1.00 | 0.0 |
| capsotu | 0.359 | 0.0166 | 1.00 | 34.8 |
| emotiondair | 0.429 | 0.132 | 1.00 | 34.1 |
| empathetic | 0.304 | 0.0300 | 1.00 | 28.2 |
| financialphrasebank | 0.641 | 0.300 | 1.00 | 48.7 |
| imdb | 0.755 | 0.510 | 1.00 | 50.0 |
| manifesto | 0.142 | 0.00957 | 1.00 | 13.4 |
| massive | 0.490 | 0.0199 | 1.00 | 48.0 |
| rottentomatoes | 0.738 | 0.509 | 1.00 | 46.6 |
| trueteacher | 0.342 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.736 | 0.511 | 1.00 | 46.0 |
| wikitoxic_obscene | 0.787 | 0.514 | 1.00 | 56.3 |
| wikitoxic_threat | 0.400 | 0.360 | 1.00 | 6.3 |
| wikitoxic_toxicaggregated | 0.746 | 0.508 | 1.00 | 48.4 |
| yahootopics | 0.414 | 0.0647 | 1.00 | 37.4 |
| yelpreviews | 0.860 | 0.513 | 1.00 | 71.1 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6240
- ece
- 0.1259
- accuracy
- 0.6623
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe weights were trained during the session from the declared base, answerdotai/ModernBERT-base at the pinned revision 8
The weights were trained during the session from the declared base, answerdotai/ModernBERT-base at the pinned revision 8949b909. train.py fine-tunes the full encoder plus a LayerNorm/Linear head with AdamW. calibrate.py exports encoder/, head.safetensors and final_config.json. The trajectory records the training chain: run1, then run2, then run4 (resumed from run2 best.pt), then run5 (resumed from run4 best). Run3 hit a CUDA OOM, was aborted, and none of its weights were used. The run5 export was copied into output/model with scp (trajectory#169). The training logs and manifest list the trained weights and declare no teacher models or reused artifacts. The head inventory (logit_scale, norm, proj) matches model_def.py.
✓no evaluation data useThe data prep scripts load only train splits of declared public non-excluded datasets, each with a recorded revision. Th
The data prep scripts load only train splits of declared public non-excluded datasets, each with a recorded revision. These include dbpedia_14 train, 20NG, TREC, CLINC, GoEmotions, TweetEval, OLID, Davidson, fintweet, the NLI corpora, tasksource/dynahate and SetFit/hate_speech18 (trajectory#114, trajectory#116). prep3.py only reuses v2 train rows. No step uses a test or evaluation split, or BTZSC data, for training or calibration.
✓no public split tuningCheckpoint selection and temperature calibration were done on dev, dev_v2 and dev_v3, which are held out from the train
Checkpoint selection and temperature calibration were done on dev, dev_v2 and dev_v3, which are held out from the train splits (trajectory#169, trajectory#141). Looking at the head of the public validation.jsonl in step 3 was inspection only. The frontier selfcheck sent public inputs as a throughput gate and reports no score (trajectory#120). The v3 'clean text is not toxic' augmentation came from probes on handwritten sentences (trajectory#143, trajectory#145), not from public-split results.
✓no jev reproductionsThe model is an in-session fine-tune of ModernBERT-base with a custom dual-encoder head defined in model_def.py. The man
The model is an in-session fine-tune of ModernBERT-base with a custom dual-encoder head defined in model_def.py. The manifest lists no teacher models or reused artifacts. No external zero-shot classifier checkpoint was downloaded, distilled or reproduced.
✓stateless zero shot inferenceFor every request, serve.py encodes the text and its candidate labels fresh. It mean-pools, projects, normalizes and ret
For every request, serve.py encodes the text and its candidate labels fresh. It mean-pools, projects, normalizes and returns a temperature-scaled softmax. It keeps no cache across requests, makes no external calls and does no test-time adaptation. Dynamic batching only groups concurrent requests.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline and runs serve.py with MODEL_DIR set to the shipped model directory. serve
serve.sh installs the vendored wheels offline and runs serve.py with MODEL_DIR set to the shipped model directory. serve.py loads AutoModel and the tokenizer from model/encoder, the norm/proj/logit_scale tensors from model/head.safetensors, and the temperature from model/final_config.json. These are the inventoried files the trajectory shows coming from the run5 export (trajectory#169).
✓harness credentials used only by the harnessI found no trajectory step that reads, prints or misuses harness API credentials. HF_TOKEN was passed only to the partic
I found no trajectory step that reads, prints or misuses harness API credentials. HF_TOKEN was passed only to the participant's own GPU box for dataset downloads, and the matches for API_KEY and BASE_URL are only in the spin help text. serve.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 14m into the runtook 3m 40s
startup 82.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 82.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0025780059603826766, 0.003339519112842656, 0.9539488206715947, 0.004059450806737526, 0.002297675064145496, 0.008834253295659173, 0.002596918794059871, 0.0014628607695550673, 0.0027967154312022327, 0.0035627156959437203, 0.00348292486 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 13.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.09, p95 0.10
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed5h 53m into the runtook 2m 37s
startup 67.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 67.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.0013005531960518738, 0.0018416878116193699, 0.9764536520858248, 0.001966157442648823, 0.0012631658337470325, 0.003931945615900529, 0.0011323052344442807, 0.0007205377842385185, 0.0014367420081656678, 0.0024014832210112014, 0.00204713 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 11.6 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.08, p95 0.09
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The solution in /workspace/output/ is final.
What was built
- A shared-encoder dual-encoder zero-shot classifier (model_def.py, served by serve.py): mean-pooled ModernBERT-base (
answerdotai/ModernBERT-base@8949b909…) → LayerNorm → Linear → L2-normalized embeddings; probabilities are softmax over scaled cosine similarity to each request's label descriptions, encoded fresh per request (no cross-request caching), with dynamic batching across concurrent requests. - Training (train.py, training_config.json): classification records optimized with cross-entropy over each record's candidate label-description set, plus NLI records with masked in-batch cross-entropy. Five runs with checkpoint selection only on dev carved from training splits; the shipped run5 checkpoint scored macro-F1 0.787 / ECE 0.012 / 0.777 combined on the final train-split dev.
- Calibration: a single global temperature 0.94, fitted on train-split dev only (NLL fit and score-grid agreed), in calibrate.py.
Data policy compliance
- Only train splits of non-excluded datasets were used (DBpedia-14, 20 Newsgroups, TREC, CLINC-150, GoEmotions, TweetEval, OLID, Davidson, DynaHate, hate_speech18, financial-tweet sentiment, plus MNLI/SNLI/ANLI/WANLI/FEVER-NLI/VitaminC), with paraphrased label descriptions, programmatic negation pairs, and clean-text not-toxic augmentation. No split of any BTZSC source, no val/test split of any public dataset, and no DBpedia test-split validation rows were trained, calibrated, or selected on. Full provenance with revisions and content hashes is in training_manifest.json, architecture.json, prep_report.json, and training.log.
Verification observed
frontier selfcheckpassed twice on the real grading path (final run: startup 67.7s of 300s; earlier identical serving stack measured 323 documents/s vs the 50/s gate, all 4,400 responses valid).- Local probes confirmed sensible behavior across sentiment, topic, emotion, intent, financial sentiment, and toxicity (including clean formal text correctly classified as non-toxic after the v3 fix).
Compute cost: $14.85 of the Spin budget (single A100 lease, now stopped).