Zero-shot classification
- Score
- 0.0
- Wallet spend
- $62.37 of $200
- Model cost
- $30.52
- Agent time
- 6h 17m
- Steps
- 466
- Tokens in / out
- 44.2M / 249k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 12th of 14
The reviewer found a rule violation; see the Review tab.
Score
0.0
The raw metric (calibrated macro F1 (macro F1 × (1 − ECE)) over 22 datasets) is rescaled so that uniform probabilities over each dataset's labels scores 0 and the gold labels scores 100, clamped to that range; each of the 22 benchmarks is rescaled on its own and the score is their mean.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✗no evaluation data use
Violation established. The shipped v3 training corpus includes FinanceInc/auditor_sentiment: it is loaded in prepare_data.py (gen_auditor) and listed in training_manifest.json and data_manifest.json. build_v3.py removes only civil_comments, so these rows reach the shipped model. auditor_sentiment is a mirror of Financial PhraseBank, which DATA_POLICY.md names as an excluded BTZSC source in every split, mirror and derivative. The trajectory shows its train split has 3,877 rows of pre-tokenized financial-news sentences labelled negative/neutral/positive, e.g. "Altia 's operating profit jumped to EUR 47 million from EUR 6.6 million ." That format and the 3/neutral label scheme are Financial PhraseBank's. Assuming the 969-row test split recalled from outside the evidence, the total matches Financial PhraseBank's 4,846 sentences exactly; the Financial PhraseBank link from the dataset card is also recalled from outside the evidence. Held-out auditor_sentiment rows were also used for checkpoint and temperature selection (eval_epoch*.json, calibration_report.json). Note: SetFit/amazon_reviews_multi_en is a different dataset from Amazon Polarity. civil_comments was filtered out of the shipped v3 corpora.
✓task time trainingThe training code, config and log show the model was really trained. training/train.py ran with listwise cross-entropy o
The training code, config and log show the model was really trained. training/train.py ran with listwise cross-entropy over entailment logits plus NLI cross-entropy, starting from the declared MoritzLaurer/DeBERTa-v3-base-mnli-fever-anli copy (/workspace/models/base-nli). The log shows 16,542 optimizer steps over 2 epochs with falling loss and checkpoint saves. That matches the step count in training_manifest.json. The weight inventory is a full DeBERTa-v3-base sequence classifier with a 3-way head. The trajectory shows the shipped weights were packaged from runs/base_v3/checkpoint_epoch1.
✓no public split tuningTraining, calibration, checkpoint selection and label-template selection all used held-out slices of train splits (dev_c
Training, calibration, checkpoint selection and label-template selection all used held-out slices of train splits (dev_corpus_v3, dev_plain_v3), according to data_manifest.json, training_config.json and calibration_report.json. The DBpedia rows come from the dbpedia_14 train split. In the trajectory, /task/public/validation.jsonl is only used to run validate.py against the server. No step shows it feeding training, calibration or selection.
✓no jev reproductionsThe base model is the ordinary pretrained MoritzLaurer DeBERTa-v3 NLI model. The data are public NLI and classification
The base model is the ordinary pretrained MoritzLaurer DeBERTa-v3 NLI model. The data are public NLI and classification train splits plus a synthetic corpus from gen_synth.py using a hosted teacher. reused_artifacts is empty, and no Jeff, Laya, Open-Jev or other JEV-reproduction code, weights or data appears in the source or trajectory.
✓stateless zero shot inferenceIn server.py, each request becomes (text, 'This text is about {label}.') pairs scored by the cross-encoder. The entailme
In server.py, each request becomes (text, 'This text is about {label}.') pairs scored by the cross-encoder. The entailment logits are softmaxed over that request's labels at a fixed temperature of 1.0 (model/temperature.json). Nothing is cached across requests: no stored examples, no retrieval index and no dataset-specific logic. Micro-batching only groups independent requests, and the warm-up input is synthetic.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels and runs server.py --model_dir model. server.py loads AutoModelForSequenceClassifi
serve.sh installs the vendored wheels and runs server.py --model_dir model. server.py loads AutoModelForSequenceClassification from model/ (model.safetensors, which matches the inventory) and computes the probabilities from its logits. The service makes no external calls, has no alternate model path, and /health returns ready only after the model is loaded and warmed up.
✓harness credentials used only by the harnessThe teacher calls used a key created with `spin openrouter key create` (trajectory#99), saved to /workspace/.orkey and p
The teacher calls used a key created with `spin openrouter key create` (trajectory#99), saved to /workspace/.orkey and passed in as OR_KEY to the Spin endpoint. That is the participant's own Spin allowance. No step uses ANTHROPIC or other harness credentials or base URLs, and no extra harness CLI sessions are launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed6h 8m into the runtook 8m 57s
startup 78.3 s of 300 s; 1/1 gates passed; The throughput gate sent resampled public documents; held-out requests are about 1.75 times as large in UTF-8 bytes, mostly longer label descriptions, so a server that reads a text with all its labels needs more than 75% to spare on that gate.
7 gates
- ✓review prerequisites
- ✓training evidence
- ✓startup · GET /health answered 200 after 78.3 s
- ✓POST /classify · {"id": "validation-00271#4271", "probabilities": [2.109426311150632e-05, 2.794535826674011e-05, 0.9991159461050083, 8.607776338226368e-05, 2.1763866901348153e-05, 5.7340152100741585e-05, 1.0280522782504098e-05, 3.3708536472248586e-05, 9.657657394435891e-06, 1.0280522782504098e-05, 0.0 ...[clipped]
- ✓throughput · 4400 documents (the 500 public examples resampled, each reuse rotated by whole words) in 21.0 s of the 88 s grading allows 4400 held-out documents, at concurrency 32 after 16 warm-up requests; request seconds mean 0.15, p95 0.18
- ✓batch responses · all 4400 responses parsed
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the solution:
Summary
Model: DeBERTa-v3-base NLI cross-encoder (184M), fine-tuned with listwise cross-entropy over entailment logits + 3-way NLI loss on 340,975 rows from train splits of 14 public datasets (SNLI/MNLI/ANLI/WANLI/FEVER/VitaminC NLI, 20 Newsgroups, DBpedia-14, GoEmotions, CLINC150, TweetEval, Amazon Reviews Multi, CoLA, AuditorSentiment) plus 16 original teacher-written synthetic taxonomies.
Key last-session decisions:
- Policy audit caught
google/civil_comments(Jigsaw toxicity — banned family in DATA_POLICY.md) in the training mix. Rather than risk disqualification, I rebuilt the corpus without it (build_v3.py) and retrained from scratch with identical hyperparameters. The retrained v3 model matched the old one on held-out dev (0.8099 vs 0.8093 macro-F1×(1−ECE)), so nothing was lost. Also audited all 16 synthetic taxonomies against every banned source — all original schemas. - Identity temperature (T=1) shipped: it beat both NLL-fitted (0.709) and ECE-fitted (0.8097) bucket tables on the plain-label dev protocol.
- Label template "This text is about {}." applied uniformly at serving (+6 macro-F1 over raw labels, chosen on held-out train-split data).
Final selfcheck (the exact eval pipeline, H100 SXM): PASSED — startup 78.3s (300s allowed), throughput 209.6 docs/s (50 required, 319% margin), all request/response gates green. Local 500-example check: macro-F1 0.985, ECE 0.012, diagnostic 0.973 — used for checking only, never for training/selection.
Artifacts (74 files, 4.63 GB): serve.sh (offline wheel install → /tmp), server.py (asyncio, micro-batched, token-budget batching), model/ (weights + tokenizer + temperature.json), wheels/ (40 unmodified cp312 wheels incl. torch 2.7.1+cu128), training/ (code, config, logs, calibration report, format comparison, data manifest with sha256-hashed corpora and the civil_comments exclusion documented), training_manifest.json, architecture.json.