Data-efficient pretraining
- Score
- 6.9
- Wallet spend
- $10.47 of $50
- Model cost
- $0.70
- Agent time
- 2h 22m
- Steps
- 228
- Tokens in / out
- 22.1M / 234k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 13th of 19
Score
6.9raw metric 0.06928
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4464
- reward
- 0.0693
- nll_nats
- 5176846.4328
- bytes
- 5163651
- documents
- 551
- tokens
- 1256036
- unscored_documents
- 0
- vocab_size
- 32768
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In model.py, GPT bu
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In model.py, GPT builds every weight with nn.init.normal_ (std 0.02, with scaled residual projections). train.py's train_one uses a fresh GPT(cfg) for each seed. The BPE tokenizer is trained from an empty models.BPE() using BpeTrainer over the corpus files (tokenizer_utils.train_tokenizer). There is no from_pretrained or hub download, and no checkpoint is loaded during training. The --reuse-tokenizer and --data-cache options are off by default and train.sh does not pass them.
✓no embedded external dataThe shipped files are code, configuration and logs. tokenizer_utils.py holds only the GPT-2 bytes_to_unicode algorithm,
The shipped files are code, configuration and logs. tokenizer_utils.py holds only the GPT-2 bytes_to_unicode algorithm, which builds byte ranges from code and contains no vocabulary or merges. No weights, vocabularies, word lists or tables are embedded. The vocabulary, merges and weights are all produced from CORPUS_DIR when train.py runs.
✓training uses only corpustrain.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR. The tokenizer and the token stream re
train.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR. The tokenizer and the token stream read only *.txt files under corpus_dir (_corpus_files, load_line_tokens). Outputs go only to output_dir: safetensors, config.json, vocab.json, tokenizer.json, train.log and dev_metrics.jsonl. There is no network access and no pretrained loading. The dev split is carved from the training files. data_manifest.md and architecture.json declare the corpus as BabyLM-2026-Strict training files. The trajectory steps that look this up (trajectory#79, trajectory#80) only checked provenance through the HF API and did not download data for training.
✓serving uses only replay outputsserve.sh runs serve.py. ModelServer reads config.json, tokenizer.json, model_*.safetensors and vocab.json only from WEIG
serve.sh runs serve.py. ModelServer reads config.json, tokenizer.json, model_*.safetensors and vocab.json only from WEIGHTS_DIR, and all of these are written by train.py. It also imports the submitted model.py, scoring.py and tokenizer_utils.py. It does not use the network, the corpus or any other model.
✓causal normalized scoring/score and /dist both use the same loaded ensemble. Results are combined with logsumexp minus log K (serve.py _combine).
/score and /dist both use the same loaded ensemble. Results are combined with logsumexp minus log K (serve.py _combine). The causal transformer uses SDPA with is_causal=True. scoring.py picks windows by absolute position, so /dist (next_token_logprobs) uses the same window as /score for every prefix. /dist returns a log_softmax over the full vocabulary and is renormalized after ensembling. There is no request caching, no lookup of previously seen text and no special-casing of endpoints.
✓no slowrun lookupThe only Slowrun mentions in the trajectory are the task prompt (trajectory#1) and reads of /task/CONTRACT.md (trajector
The only Slowrun mentions in the trajectory are the task prompt (trajectory#1) and reads of /task/CONTRACT.md (trajectory#124, trajectory#216), which are allowed. No search, fetch, clone or other access to qlabs-eng/slowrun or pages about it was found.
✓harness credentials used only by the harnessNo trajectory step uses ANTHROPIC/OPENAI/OPENROUTER keys or the harness base URL, and no other harness CLI is launched.
No trajectory step uses ANTHROPIC/OPENAI/OPENROUTER keys or the harness base URL, and no other harness CLI is launched. Network use was limited to the Spin CLI, an s5cmd binary download, and HF API listing calls with HF_TOKEN (trajectory#79, trajectory#80), which is allowed read-only use.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 4m into the runtook 4m 57s
startup 8.1 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 121 s, writing 10 files (2205278584 bytes) to OUTPUT_DIR. End of its log: bedding 84.95M) [20:39:23] [m4] probe 0.292s/step 224568 tokens/s [20:39:23] [m4] planned steps=82 (est 0.4 min) [20:39:24] [m4] step 1/82 loss 10.5330 ema 10.5330 lr 1.92e-09 tok/s 221827 elapsed 0s [20:39:29] [m4] step 20/82 loss 6.2265 ema 9.0730 lr 7.17e-04 tok/s 223562 elapsed 6s [20:39:35] [m4] step 40/82 loss 5.8797 ema 7.7114 lr 4.65e-04 tok/s 223563 elapsed 12s [20:39:41] [m4] step 60/82 loss 5.5458 ema 6.7725 lr 1.90e-04 tok/s 223546 elapsed 18s [20:39:47] [m4] step 80/82 loss 5.1690 ema 6.1883 lr 5.06e-05 tok/s 223539 elapsed 23s [20:39:47] [m4] training done steps=82 (planned 82, interrupted=False) 24.0s [20:39:48] [m4] saved model_4.safetensors (114.0s elapsed total) [20:39:48] no room for another model (est 26s, 3s left): stop at 5 [20:39:49] [m0] FINAL dev_bpb 2.5522 [20:39:50] [m1] FINAL dev_bpb 2.3937 [20:39:51] [m2] FINAL dev_bpb 2.2701 [20:39:52] [m3] FINAL dev_bpb 2.2672 [20:39:52] [m4] FINAL dev_bpb 2.1813 [20:39:52] saved config/vocab (4.3s); total elapsed 118.3s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 8.1 s
- ✓GET /vocab · {"tokens": 32768, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": -4.440892098500626e-16, "score_minus_dist": -8.505659288005063e-09}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 20.6 s and 86 of 516 /dist and truncation requests in 6.6 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (12 s) as measured
- ✓slowest request · 183 answered requests, median 0.69 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1980 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed2h 13m into the runtook 3m 14s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 121 s, writing 6 files (443317772 bytes) to OUTPUT_DIR. End of its log: p 160/306 loss 4.5915 ema 4.7788 lr 4.12e-04 tok/s 220588 elapsed 48s [21:47:14] [m0] step 180/306 loss 4.3699 ema 4.5999 lr 3.35e-04 tok/s 220645 elapsed 53s [21:47:20] [m0] step 200/306 loss 4.4622 ema 4.5743 lr 2.61e-04 tok/s 220684 elapsed 59s [21:47:26] [m0] step 220/306 loss 4.5052 ema 4.5379 lr 1.94e-04 tok/s 220712 elapsed 65s [21:47:32] [m0] step 240/306 loss 4.3272 ema 4.4580 lr 1.37e-04 tok/s 220700 elapsed 71s [21:47:38] [m0] step 260/306 loss 4.1458 ema 4.3590 lr 9.27e-05 tok/s 220722 elapsed 77s [21:47:44] [m0] step 280/306 loss 3.9187 ema 4.3151 lr 6.29e-05 tok/s 220734 elapsed 83s [21:47:49] [m0] step 300/306 loss 4.2279 ema 4.2653 lr 4.90e-05 tok/s 220753 elapsed 89s [21:47:51] [m0] training done steps=306 (planned 306, interrupted=False) 90.8s [21:47:52] [m0] saved model_0.safetensors (115.8s elapsed total) [21:47:52] no room for another model (est 95s, 2s left): stop at 1 [21:47:54] [m0] FINAL dev_bpb 1.8268 [21:47:54] saved config/vocab (2.1s); total elapsed 117.9s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 32768, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": 1.5172990197598324e-08, "score_minus_dist": 1.430511474609375e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.5 s and 86 of 516 /dist and truncation requests in 2.5 s, each scaled to its full count, plus the first requests (0.5 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.26 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5071 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-20 21:55:08 UTC.
Final submission — 12 files, 110.3 KB, all 8 contract gates passing:
| Piece | What it does |
|---|---|
train.sh / train.py | Trains byte-level BPE (32,768) + as many independent 110M GPT models as fit in the budget; deadline-driven so it always exits inside 3600 s |
serve.sh / serve.py | Loads all saved members and serves the ensemble average of next-token distributions |
architecture.json, data_manifest.md, logs/ | Architecture, measured numbers, corpus provenance (BabyLM-2026-Strict *.train.txt only), rehearsal log (633 lines) |
Measured results (one H100 SXM):
- Full 3600 s rehearsal: 3434.7 s, 8 models × 1422 steps (5 passes over the 15.5M-token stream, 0.293 s/step).
- Held-out dev (carved from the training files only): single model 1.3986 bpb → 8-model ensemble 1.3073 bpb; curve 1→8: 1.3986, 1.3511, 1.3374, 1.3274, 1.3211, 1.3118, 1.3079, 1.3073.
- Contract test with the real ensemble: max |score−dist| 2e-6 nats, max |logsumexp| 5e-16, truncation drift 2e-6 nats.
- Public validation sample (one-time report after all decisions were frozen, disclosed in the manifest; no decision used it): 1.4621 bpb over 350,116 bytes.
One note on the final selfcheck: its 240 s probe budget minus the 120 s reserve leaves ~90 s, so it trained a single 306-step model — expected behavior of the adaptive loop, not the grading path. The 3600 s rehearsal is what validates the 8-model run, and the shipped train.py differs from the rehearsed one only by fail-safe guards (OOM batch fallback, save retry, eval try/except).