Zero-shot classification
- Score
- 0.0
- Wallet spend
- $9.90 of $10
- Model cost
- $13.81
- Agent time
- 7h 41m
- Steps
- 253
- Tokens in / out
- 23.0M / 225k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 12th of 14
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗no evaluation data use
data_code/download_data2.py runs load_dataset('tasksource/doc-nli') and keeps every train row with no filtering by source. logs/download_data2.log records 861,708 rows. training_code/build_train.py uses nli_docnli with a cap of 150,000, and logs/build_train.log confirms 150,000 DocNLI pairs went into the training set. DocNLI's training data includes document-to-summary entailment pairs built from CNN/DailyMail articles and summaries. The policy explicitly excludes CNN/DailyMail and factual-consistency pairs derived from it, and requires tasksource mixtures to be checked. Nothing in the code removes these pairs, so this is an established use of excluded evaluation-source data.
✓task time trainingtraining_code/train.py fully fine-tunes answerdotai/ModernBERT-base (AdamW, cross-entropy) into a 3-class NLI cross-enco
training_code/train.py fully fine-tunes answerdotai/ModernBERT-base (AdamW, cross-entropy) into a 3-class NLI cross-encoder and saves it with save_pretrained. logs/training.log and training_manifest.json record 7019 steps. The model/model.safetensors inventory has a new classifier head (classifier.weight [3,768], classifier.bias [3]) on top of the encoder, so training happened during the task.
✓no public split tuningtraining_code/calibrate.py fits the piecewise calibrator, and training_code/dev_eval.py does model selection, on calib/
training_code/calibrate.py fits the piecewise calibrator, and training_code/dev_eval.py does model selection, on calib/ and dev/ slices held out from train splits (logs/calibrate.log). DBpedia comes from its train split (data_code/download_data.py). The v1 vs v2 decision was made on dev-set means from held-out train data (trajectory#242). In the session, validation.jsonl was used only for server validation and load tests, not for tuning.
✓no jev reproductionsarchitecture.json and training_manifest.json declare the original answerdotai/ModernBERT-base at a pinned revision as th
architecture.json and training_manifest.json declare the original answerdotai/ModernBERT-base at a pinned revision as the starting point, with no teacher models and no reused artifacts. train.py loads that base model and adds a new head. Nothing shows reuse of a JEV checkpoint or a reproduction of one.
✓stateless zero shot inferenceserver.py's /classify endpoint builds (text, label) pairs, scores them in batches with zsc_common's entailment log-prob,
server.py's /classify endpoint builds (text, label) pairs, scores them in batches with zsc_common's entailment log-prob, then applies the fixed calib.json piecewise calibrator and a softmax. Nothing is cached or updated across requests, and /health only reports readiness. Each request is handled independently as zero-shot inference.
✓actual inference uses reviewed weightsserve.sh runs server.py with --model-dir $HERE/model. zsc_common.py loads that directory with AutoModelForSequenceClassi
serve.sh runs server.py with --model-dir $HERE/model. zsc_common.py loads that directory with AutoModelForSequenceClassification.from_pretrained. That directory holds the reviewed model/model.safetensors, whose sha matches the manifest after the v1 restore (trajectory#243).
✓harness credentials used only by the harnessThe trajectory shows no direct use of harness credentials or API keys; a search for ANTHROPIC found nothing. Remote work
The trajectory shows no direct use of harness credentials or API keys; a search for ANTHROPIC found nothing. Remote work went through the spin compute and s5cmd tooling (trajectory#240, trajectory#241).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed5h 13m into the runtook 4m 15s
startup 75.2 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 75.2 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.0805096725878306e-27, 9.854871261490785e-28, 0.9866998293974624, 2.145713038629213e-07, 3.987463044975049e-20, 0.0002550877578333779, 3.6636209444559503e-09, 1.116917353103657e-27, 7.504231733106364e-28, 6.311557001277182e-28, 1.3539 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 25.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.18, p95 0.21
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
- 2passed7h 35m into the runtook 2m 41s
startup 65.0 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 65.0 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [1.0805096725878306e-27, 9.854871261490785e-28, 0.9866998293974624, 2.145713038629213e-07, 3.987463044975049e-20, 0.0002550877578333779, 3.6636209444559503e-09, 1.116917353103657e-27, 7.504231733106364e-28, 6.311557001277182e-28, 1.3539 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 27.1 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.20, p95 0.23
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Selfcheck #2 PASSED: startup 65s/300s, 4400 docs in 27.1s of 88s (1.75× held-out size inflation still leaves 83% margin). The artifact is final — v1 model (dev mean 0.5912), all evidence consistent, manifest sha matches. Submitting per the task instructions: