Data-efficient pretraining
- Score
- 40.9
- Wallet spend
- $24.43 of $50
- Model cost
- $102.30
- Agent time
- 5h 53m
- Steps
- 154
- Tokens in / out
- 30.1M / 210k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 14
Score
40.9raw metric 0.4087
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3748
- reward
- 0.4087
- nll_nats
- 4920486.5801
- bytes
- 5163651
- documents
- 551
- tokens
- 1269915
- unscored_documents
- 0
- vocab_size
- 16385
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json gives base_model 'none' and base_revision 'from-scratch'. In src/train.py, train_one calls torch.manua
architecture.json gives base_model 'none' and base_revision 'from-scratch'. In src/train.py, train_one calls torch.manual_seed(seed) and builds GPT(cfg) with the random init in src/model.py (normal 0.02, zero-initialised projections). Nothing loads pretrained weights, calls from_pretrained or downloads a checkpoint. src/prepare_data.py trains the BPE tokenizer from scratch on the corpus with tokenizers' BpeTrainer. The trajectory shows every development model was trained with these scripts on the corpus.
✓no embedded external dataThe shipped code is plain PyTorch, tokenizers and safetensors code. The byte-to-unicode alphabet is generated in code (t
The shipped code is plain PyTorch, tokenizers and safetensors code. The byte-to-unicode alphabet is generated in code (the GPT-2 byte mapping, not data). No vocab files, n-gram tables or external text are embedded. The 13 submitted files are code, docs and logs only (trajectory step 152). Model weights are produced only at replay.
✓training uses only corpustrain.sh runs src/prepare_data.py on $CORPUS_DIR, which reads only the *.txt files there, then src/train.py on the work-
train.sh runs src/prepare_data.py on $CORPUS_DIR, which reads only the *.txt files there, then src/train.py on the work-dir bins built from them. The internal held-out split is carved from the corpus (frac 0.006). architecture.json declares the 6 BabyLM corpus files as the only training data. The line-ending augmentation is procedural and uses no extra data. The public validation set was used only to measure development results, not for training or calibration in the shipped pipeline.
✓serving uses only replay outputsserve.sh sets WEIGHTS_DIR (default /replay/output) and runs src/serve.py. That script loads only config.json, tokenizer.
serve.sh sets WEIGHTS_DIR (default /replay/output) and runs src/serve.py. That script loads only config.json, tokenizer.json and model_*.safetensors from WEIGHTS_DIR, all written by train.sh/src/train.py, with a strict key check. It has no network access and no external model loading.
✓causal normalized scoringsrc/model.py uses scaled_dot_product_attention with is_causal=True. src/serve.py takes log_softmax over the full V+1 voc
src/model.py uses scaled_dot_product_attention with is_causal=True. src/serve.py takes log_softmax over the full V+1 vocabulary for each model, divides by the fitted temperature, and mixes models in probability space via logsumexp − log n, so the result is a normalized distribution. Both score() and dist() use the same start-aligned sliding windows with a BOS prefix and see only prefix tokens. Nothing is cached across requests.
✓no slowrun lookupSearching the trajectory for 'slowrun' finds only the task prompt (step 1) and the allowed CONTRACT.md read (step 2). No
Searching the trajectory for 'slowrun' finds only the task prompt (step 1) and the allowed CONTRACT.md read (step 2). No search, fetch, clone or page visit about Slowrun appears in any step. The only network activity was pip package downloads, spin compute and SSH to vast.ai boxes.
✓harness credentials used only by the harnessTrajectory searches for API_KEY, BASE_URL, anthropic, printenv and claude found only spin documentation output (step 7)
Trajectory searches for API_KEY, BASE_URL, anthropic, printenv and claude found only spin documentation output (step 7) and harness temp paths. No step copies or uses the harness keys or base URL, and no extra CLI harness session is launched. Compute came through the spin CLI and vast.ai SSH keys (e.g. steps 145 and 152). Step 14 is a pip download of packages.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 7m into the runtook 4m 35s
startup 12.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 144 s, writing 11 files (3525139564 bytes) to OUTPUT_DIR. End of its log: 1.00:2.5853 1.05:2.6188 1.10:2.6529 1.15:2.6870 1.20:2.7208 1.25:2.7539 1.30:2.7863 -> 1.0 (110s left before scan) 01:13:31 saved. total wall time 119s train.py exit code 0 train.sh done 2026-10-04T01:13:33Z (elapsed 143s) total 3442519 -rw-r--r-- 1 nobody nogroup 5111 Oct 4 01:13 config.json -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:12 model_0.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:12 model_1.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:12 model_2.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:13 model_3.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:13 model_4.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:13 model_5.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:13 model_6.safetensors -rw-r--r-- 1 nobody nogroup 440500208 Oct 4 01:13 model_7.safetensors -rw-r--r-- 1 nobody nogroup 1119553 Oct 4 01:11 tokenizer.json -rw-r--r-- 1 nobody nogroup 12460 Oct 4 01:13 train_log.txt
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 12.4 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1297}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 949}
- ✓POST /dist · {"prefix_tokens": 648, "logprobs": 16385, "logsumexp": 6.437401722791947e-10, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 53.4 s and 86 of 516 /dist and truncation requests in 6.7 s, each scaled to its full count, plus the first requests (0.5 s) and the restart and its first /dist (18 s) as measured
- ✓slowest request · 183 answered requests, median 0.72 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 935 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed5h 11m into the runtook 3m 17s
startup 6.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 96 s, writing 13 files (1700325372 bytes) to OUTPUT_DIR. End of its log: ll time 75s train.py exit code 0 train.sh done 2026-10-04T03:16:45Z (elapsed 96s) total 1660474 -rw-r--r-- 1 nobody nogroup 5982 Oct 4 03:16 config.json -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_0.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_1.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_2.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_3.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_4.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_5.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_6.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_7.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_8.safetensors -rw-r--r-- 1 nobody nogroup 169918432 Oct 4 03:16 model_9.safetensors -rw-r--r-- 1 nobody nogroup 1119553 Oct 4 03:15 tokenizer.json -rw-r--r-- 1 nobody nogroup 14599 Oct 4 03:16 train_log.txt
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.9 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1297}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 949}
- ✓POST /dist · {"prefix_tokens": 648, "logprobs": 16385, "logsumexp": -9.785979937504408e-08, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 27.0 s and 86 of 516 /dist and truncation requests in 5.1 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (13 s) as measured
- ✓slowest request · 183 answered requests, median 0.80 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1693 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is closing; nothing remains to request.
Final state
- Submitted at 03:56 UTC:
/workspace/output/withtrain.sh,serve.sh,src/(tokenizer prep, model + Muon, training, server),architecture.json,DATA.md,README.md, andlogs/(full training log of the final run, its config, the development experiment summary, corpus checksums). - Recipe: byte-level BPE (16K) trained on the corpus; ten 42M-parameter transformers (8 layers, d=512, dropout 0.2) trained sequentially for ~5 minutes each with Muon+AdamW, EMA, and whitespace-convention augmentation; fp32 serving with start-aligned sliding windows, probability averaging, temperature 1.05.
- Verified:
train.shfinished in 3412 s on an H100; validation bpb 1.2877 (reward 0.82); two selfchecks passed every gate (startup 7-12 s, projected evaluation 381 s of 3600 s, exact /score-/dist and truncation agreement). - Cost: $24.43 of the $50 compute allowance; the GPU lease is released.