Zero-shot classification
- Score
- 0.0
- Wallet spend
- $0.00 of $50
- Model cost
- $31.56
- Agent time
- 1h 46m
- Steps
- 2,088
- Tokens in / out
- 292.4M / 805k
- Self-checks
- 1 passed of 3 used
- Rank at $50 wallet
- 12th of 14
Trusted throughput client failed: TimeoutError: Whole-batch deadline exceeded
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓task time trainingtrain.py loads Alibaba-NLP/gte-modernbert-base at the pinned revision e7f32e3c. It freezes every layer except the last e
train.py loads Alibaba-NLP/gte-modernbert-base at the pinned revision e7f32e3c. It freezes every layer except the last encoder layer and runs 100 AdamW steps of symmetric InfoNCE (temperature 0.05) on 800 MultiNLI train entailment pairs. It then saves the full state_dict to model.safetensors. training.log records this run on 2026-10-03 (5,015,040 trainable parameters, losses logged every 10 steps) and the saved SHA256 597f45d9.... training_manifest.json records the same hash. The trajectory shows train.py being written (#709) and run with stdout going to training.log (#710). I found no later step that replaces model.safetensors. The weight inventory lists the ModernBERT tensors in F32, which matches a save from the fp32 training run.
✓no evaluation data useThe only training or calibration corpus is nyu-mll/multi_nli, train split. architecture.json, training_config.json and t
The only training or calibration corpus is nyu-mll/multi_nli, train split. architecture.json, training_config.json and training_manifest.json declare it at commit da70db2a with a file sha256, and train.py and training.log load it. MultiNLI is not a BTZSC evaluation source. The temperature is fixed at 1/0.05 and is not fitted on any data. I found no evaluation data in the training code or the training records.
✓no public split tuningIn the trajectory, the public validation.jsonl was used only to look at the format and at label and length statistics, f
In the trajectory, the public validation.jsonl was used only to look at the format and at label and length statistics, for latency and throughput tests, and for two validate.py runs (trajectory#830 and #1607). TEMPERATURE_SCALE = 20.0 in server.py equals 1/0.05, the training temperature in train.py. It appears in the first server version and was never changed. No hyperparameter, checkpoint or prompt was chosen on public-split scores.
✓no jev reproductionsThe base is the public Alibaba-NLP/gte-modernbert-base, built on ModernBERT-base, at pinned revisions in architecture.js
The base is the public Alibaba-NLP/gte-modernbert-base, built on ModernBERT-base, at pinned revisions in architecture.json. training_manifest.json lists no teacher models and no reused artifacts. train.py is self-contained contrastive fine-tuning code that loads only the declared base model and MultiNLI. Nothing reproduces or distills a JEV-family system.
✓stateless zero shot inferenceserver.py scores each request on its own: it embeds the current text and its supplied candidate labels with CLS pooling,
server.py scores each request on its own: it embeds the current text and its supplied candidate labels with CLS pooling, L2 normalisation, cosine similarity × 20 and a softmax. Dynamic batching only groups concurrent requests, and its label deduplication does not change any request's result. There is no cache across requests, no retrieval, no external calls and no learning at inference time.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline (--no-index) and starts uvicorn server:app. In the lifespan step, server.p
serve.sh installs the vendored wheels offline (--no-index) and starts uvicorn server:app. In the lifespan step, server.py builds a ModernBertModel from the shipped config.json and loads the shipped model.safetensors with strict=True. Every request is scored by that model, and no other model or external service is called. The weight inventory shows a full ModernBERT tensor set matching config.json.
✓harness credentials used only by the harnessThe only trajectory matches for API_KEY/BASE_URL are in the documentation the participant read at the start of the sessi
The only trajectory matches for API_KEY/BASE_URL are in the documentation the participant read at the start of the session. The participant's commands were limited to local training, serving and tests, an allowed spin s5cmd check and copying files into the output (e.g. trajectory#710, #779). No step uses harness credentials or starts an extra model session with them.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed44m 55s into the runtook 3m 48s
startup 68.7 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 68.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.017946526675619582, 0.035893053351239164, 0.14161884995727017, 0.15040898547185935, 0.03809058722988646, 0.06299597118788915, 0.09717983152240264, 0.036869735075082406, 0.011598095470638505, 0.029666707361738494, 0.1816628006348431, ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 2failed1h 1m into the runtook 4m 17s
startup 90.7 s of 300 s; 0/1 gates passed; failed throughput: 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded); The throughput gate sent resampled public documents; held-out requests ar
5 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 90.7 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.017946526675619582, 0.035893053351239164, 0.14161884995727017, 0.15040898547185935, 0.03809058722988646, 0.06299597118788915, 0.09717983152240264, 0.036869735075082406, 0.011598095470638505, 0.029666707361738494, 0.1816628006348431, ...[clipped]
- ✗throughput · 4400 public documents at concurrency 32 did not finish within the 88 s grading allows 4400 held-out documents (TimeoutError: Whole-batch deadline exceeded)
- 3passed1h 40m into the runtook 4m 22s
startup 94.5 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 94.5 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [0.017946526675619582, 0.035893053351239164, 0.14161884995727017, 0.15040898547185935, 0.03809058722988646, 0.06299597118788915, 0.09717983152240264, 0.036869735075082406, 0.011598095470638505, 0.029666707361738494, 0.1816628006348431, ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 83.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.60, p95 1.40
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
The zero-shot text classification solution has been finalized, validated via frontier selfcheck, and formally submitted.
Summary of Completed Deliverables
-
Architecture & Foundation Model:
- Base Model:
Alibaba-NLP/gte-modernbert-base(ModernBertModel), featuring 8192 context window, unpadding/SDPA attention kernel support, and strong embedding representations. - Fine-Tuning: Trained on MultiNLI entailment pairs with InfoNCE contrastive loss (
temperature=0.05), calibrating representation geometry for generalized zero-shot description matching without training on BTZSC or DBpedia-14 validation splits.
- Base Model:
-
Inference Pipeline & Optimizations:
- Server Architecture: Async Starlette web service with
uvloop,httptools, and a dedicatedThreadPoolExecutorworker to ensure non-blocking HTTP request processing under high concurrency. - Dynamic Batching: 10 ms wait window dynamically accumulating up to batch size 32.
- In-Batch Deduplication: Eliminates redundant label encoding across concurrent requests, reducing label forward passes dramatically.
- SDPA & Dynamic Sequence Padding: Uses PyTorch SDPA FlashAttention kernels with dynamic sequence truncation/padding (
max_length=384for texts,max_length=64for label descriptions), removing quadratic compute over padded tokens. - Warmup: Startup routine primes SDPA kernels across batch sizes (1, 16, 32) before opening the
/healthendpoint.
- Server Architecture: Async Starlette web service with
-
Selfcheck & Gate Results (
848883722269):- Review Prerequisites:
PASSED - Training Evidence & Provenance:
PASSED - Startup Gate:
PASSED(94.5 s / 300 s allowed) - Smoke Query:
PASSED(exact probability distribution summing to 1.0) - Throughput Gate:
PASSED(52.95 documents/sec across 4,400 documents under concurrency 32, finishing in 83.1 s vs 88 s deadline) - Batch Responses & Post-Request Health:
PASSED
- Review Prerequisites:
-
Submission Artifacts in
/workspace/output/:serve.sh: Offline entrypoint executing pip install from vendored wheels and running uvicorn.server.py: High-performance asynchronous serving app.model.safetensors,config.json,tokenizer.json,tokenizer_config.json.train.py,training_config.json,training.log,training_manifest.json,architecture.json.wheels/: 65 vendored wheels (3.8 GB) cov