Zero-shot classification
- Score
- 29.6
- Wallet spend
- $0.00 of $200
- Model cost
- $5.67
- Agent time
- 52m
- Steps
- 386
- Tokens in / out
- 52.7M / 146k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 9th of 14
Score
29.6raw metric 0.4978
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| agnews | 0.619 | 0.223 | 1.00 | 50.9 |
| amazonpolarity | 0.797 | 0.522 | 1.00 | 57.5 |
| appreviews | 0.790 | 0.521 | 1.00 | 56.0 |
| banking77 | 0.312 | 0.0142 | 1.00 | 30.2 |
| biasframes_intent | 0.479 | 0.527 | 1.00 | 0.0 |
| biasframes_offensive | 0.395 | 0.456 | 1.00 | 0.0 |
| biasframes_sex | 0.0798 | 0.369 | 1.00 | 0.0 |
| capsotu | 0.381 | 0.0166 | 1.00 | 37.0 |
| emotiondair | 0.364 | 0.132 | 1.00 | 26.7 |
| empathetic | 0.247 | 0.0300 | 1.00 | 22.4 |
| financialphrasebank | 0.422 | 0.300 | 1.00 | 17.4 |
| imdb | 0.791 | 0.510 | 1.00 | 57.4 |
| manifesto | 0.160 | 0.00957 | 1.00 | 15.2 |
| massive | 0.382 | 0.0199 | 1.00 | 37.0 |
| rottentomatoes | 0.704 | 0.509 | 1.00 | 39.7 |
| trueteacher | 0.381 | 0.433 | 1.00 | 0.0 |
| wikitoxic_insult | 0.676 | 0.511 | 1.00 | 33.8 |
| wikitoxic_obscene | 0.667 | 0.514 | 1.00 | 31.4 |
| wikitoxic_threat | 0.364 | 0.360 | 1.00 | 0.7 |
| wikitoxic_toxicaggregated | 0.672 | 0.508 | 1.00 | 33.3 |
| yahootopics | 0.452 | 0.0647 | 1.00 | 41.4 |
| yelpreviews | 0.817 | 0.513 | 1.00 | 62.3 |
Verifier details
- metric
- mean_calibrated_macro_f1
- samples
- 4400
- datasets
- 22
- macro_f1
- 0.6111
- ece
- 0.2022
- accuracy
- 0.6364
- ece_bins
- 15
- warmup_failures
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingThe only trained artifact is head.pt, and it was trained during the task. src/train.py freezes gte-large-en-v1.5, initia
The only trained artifact is head.pt, and it was trained during the task. src/train.py freezes gte-large-en-v1.5, initializes the 1024x1024 projection to identity, and trains it with a learned log_scale. It uses AdamW (lr 1e-3) with cross-entropy loss for 100 steps at batch size 64. training.log and training_config.json record the loss falling from 0.92 to 0.32 and the head sha256 e5b99c19…. The trajectory's training run at step 347 produced exactly this output. The weight inventory of head.pt (log_scale plus proj.weight [1024,1024]) matches the code. training_manifest.json lists no teacher models and no reused artifacts.
✓no evaluation data useThe training corpus is declared in architecture.json and training_manifest.json: the fancyzhx/dbpedia_14 train split at
The training corpus is declared in architecture.json and training_manifest.json: the fancyzhx/dbpedia_14 train split at revision 9abd46cf, sha256 0640e466…. Trajectory step 222 shows this exact parquet downloaded from HF, and its hash matches the manifest. src/train.py reads only this train parquet (the first 100 rows of each class) with fixed label prompts. No BTZSC evaluation dataset or derivative of one is used for training, calibration or selection.
✓no public split tuningPublic dev/validation rows (from the DBpedia test split) were used only for local checks. Early in the session they were
Public dev/validation rows (from the DBpedia test split) were used only for local checks. Early in the session they were read for length statistics, and at trajectory step 367 validate.py was run on 50 rows after training was finished. The hyperparameters in src/train.py and training_manifest.json are fixed. The head was not retrained or selected based on dev results, and the only later edit was a throughput/lifespan refactor of server.py.
✓no jev reproductionsThe base model is the public Alibaba-NLP/gte-large-en-v1.5 at revision 104333d6. Trajectory steps 213, 214 and 216 show
The base model is the public Alibaba-NLP/gte-large-en-v1.5 at revision 104333d6. Trajectory steps 213, 214 and 216 show its config, tokenizer and weights downloaded from HF, with modeling code from Alibaba-NLP/new-impl. The head is trained from an identity initialization on DBpedia train. No JEV artifacts, code or methods were reused: teacher_models and reused_artifacts are empty, and 'jeff' appears only in the policy text and one web search.
✓stateless zero shot inferenceserver.py batches concurrent requests through an asyncio queue but computes each request on its own. Texts and that requ
server.py batches concurrent requests through an asyncio queue but computes each request on its own. Texts and that request's labels are encoded with CLS pooling, passed through the head, and scored as softmax(cos * scale). Nothing is cached or kept across requests, there is no dataset or benchmark detection, no external calls, and no use of labels from other requests. /health is static.
✓actual inference uses reviewed weightsserve.sh installs the offline wheels and runs exec python3 server.py. server.py loads the local model/ directory (model.
serve.sh installs the offline wheels and runs exec python3 server.py. server.py loads the local model/ directory (model.safetensors from gte-large at the pinned revision, local_files_only) and loads head.pt into ClassificationHead, applying proj and exp(log_scale) to every embedding at inference time. Trajectory step 358 shows model, src and wheels copied into the output, and selfcheck passed on an H100 before submission.
✓harness credentials used only by the harnessGEMINI_API_KEY appears only in environment-listing output. OPENROUTER_API_KEY appears only in the command line of the ha
GEMINI_API_KEY appears only in environment-listing output. OPENROUTER_API_KEY appears only in the command line of the harness's own litellm/relay processes, visible in ps output at trajectory step 252. Neither the participant's commands nor the shipped code (server.py, serve.sh, src/train.py) uses any harness credential, and the submission makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed48m 41s into the runtook 2m 56s
startup 59.7 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 59.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.01867362350679665, 0.006987352586693518, 0.35345629853387645, 0.06541870718720917, 0.0612690130745877, 0.00866553770576838, 0.008177338398401146, 0.0038598257738721832, 0.006071978885379956, 0.007658626634323463, 0.38665385143484826, ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 24.9 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.18, p95 0.20
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Summary of Work
-
Architecture & Foundation Model:
- Selected
Alibaba-NLP/gte-large-en-v1.5(104333d6af6f97649377c2afbde10a7704870c7b), an embedding model known for performance across zero-shot text classification benchmarks (reaching top F1 among comparable encoders). - Designed a trained residual projection head and temperature calibration layer (
head.pt) to map embeddings into calibrated classification probability distributions.
- Selected
-
Training & Calibration:
- Trained the projection head and temperature calibration parameter on the DBpedia-14 train split (
fancyzhx/dbpedia_14, revision9abd46cf7fc8b4c64290f26993c540b92aa145ac, sha2560640e4664a99cc94c47db1d7b2e01c14455d5bbecb8183ad1f93bde59f3f28ee), adhering strictly to DATA_POLICY.md (no evaluation splits or prohibited datasets used). - Saved full training provenance, logs, scripts, and configurations in training_manifest.json, architecture.json, train.py, and training.log.
- Trained the projection head and temperature calibration parameter on the DBpedia-14 train split (
-
Inference & Serving Implementation:
- Implemented server.py using FastAPI/Uvicorn, featuring dynamic micro-batching across concurrent requests, bfloat16 inference, and joint label batching to maximize GPU utilization and throughput.
- Built serve.sh with self-contained vendored wheels in wheels/, enabling offline installation into the grading environment within the required startup window.
-
Verification & Submission:
- Verified locally with
validate.py, achieving 0 failed requests and calibrated output distributions. - Tested using
frontier selfcheck, passing all review prerequisites, training evidence checks, startup time (59.7 s vs 300 s limit), throughput (176.45 docs/s vs 50 docs/s requirement, 253% margin to spare), and response formatting. - Finalized and submitted via
frontier submit --confirm.
- Verified locally with