Zero-shot classification
Train a fast calibrated zero-shot text classifier
TypesafeInstruction
Train a zero-shot text classifier that works with any set of label descriptions. I have provided 500 labeled examples for checking your server locally; do not train, calibrate or select on them.
Don't use any split of BTZSC or its source datasets (listed in /task/public/DATA_POLICY.md). Start from any pretrained LLM or encoder, but not one fine-tuned on those datasets, and don't use models from open-source reproductions of TypeSafe's JEV.
Write a script serve.sh which serves the classifier. It should have a POST /classify endpoint which takes a text and a list of label descriptions and returns a probability for each label. It should run offline on one H100, be ready within 300 seconds, and handle at least 50 documents per second.
Optimize macro-F1 * (1 - ECE), averaged over datasets.
Do not train, fine-tune, calibrate or select on the validation or test split of any public dataset or benchmark; training splits and other public corpora are fine.
Submit in /workspace/output/. Interface and runtime details: /task/CONTRACT.md.
Supporting files: /task/public/. Evaluate with python /task/validate.py --server-url URL.
Test the finished submission the way it will be run with frontier selfcheck.
Hand it in with frontier submit when it is final.
Use spin --help for compute and model APIs.
Trials
42 graded · click a score for the trajectoryScoring
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
| Benchmark | Scores 0 at | Scores 100 at |
|---|---|---|
| agnews | 0.2235 | 1 |
| amazonpolarity | 0.5221 | 1 |
| appreviews | 0.5214 | 1 |
| banking77 | 0.01417 | 1 |
| biasframes_intent | 0.527 | 1 |
| biasframes_offensive | 0.4559 | 1 |
| biasframes_sex | 0.3694 | 1 |
| capsotu | 0.01663 | 1 |
| emotiondair | 0.1325 | 1 |
| empathetic | 0.02997 | 1 |
| financialphrasebank | 0.3004 | 1 |
| imdb | 0.5096 | 1 |
| manifesto | 0.009569 | 1 |
| massive | 0.01988 | 1 |
| rottentomatoes | 0.5092 | 1 |
| trueteacher | 0.4335 | 1 |
| wikitoxic_insult | 0.5113 | 1 |
| wikitoxic_obscene | 0.5139 | 1 |
| wikitoxic_threat | 0.3597 | 1 |
| wikitoxic_toxicaggregated | 0.5079 | 1 |
| yahootopics | 0.06474 | 1 |
| yelpreviews | 0.5134 | 1 |