Trial · babylm-replay-2zjl2r

Data-efficient pretraining

Claude Opus 5.5xhighHarness Claude Code 2.1.282

Wallet

Score: 65.1
Wallet spend: $9.60 of $10
Model cost: $26.53
Agent time: 3h 17m
Steps: 198
Tokens in / out: 44.2M / 261k
Self-checks: 3 passed of 3 used
Rank at $10 wallet: 1st of 14

Loading trajectory…

pass6 of 6 checks passed · reviewed by claude-opus-5-5 · read 198 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. src/tok.py builds an em

architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. src/tok.py builds an empty HF tokenizers BPE model and trains it on the corpus files with BpeTrainer. In src/train.py, train_member creates a fresh M.GPT(mcfg) after torch.manual_seed and trains it with AdamW and Muon. src/model.py defines every layer itself, with plain nn.Embedding/Linear initialization and some projections set to zero. Nothing calls from_pretrained, loads a checkpoint or distills from another model. The trajectory shows no model downloads; the final self-check trained a member from scratch (trajectory#191).

architecture.jsonsrc/train.pysrc/model.pysrc/tok.pytrain.shtrajectory#191

✓no embedded external dataThe shipped source contains only code, hyperparameters and algorithm constants: a cl100k-style pre-tokenizer regex, GPT-

The shipped source contains only code, hyperparameters and algorithm constants: a cl100k-style pre-tokenizer regex, GPT-2's byte-to-unicode mapping (an algorithmic byte remapping, not learned data), and Newton-Schulz coefficients. It contains no vocabularies, merge lists, weights or text tables. The vocabulary is learned at replay time by tok.train(files,...). The submission audit found no large files, and the whole output was 231 KB of code and logs (trajectory#192, trajectory#197).

src/tok.pysrc/model.pysrc/train.pyconfig.jsonDATA.mdtrajectory#192

✓training uses only corpustrain.sh runs src/train.py, which reads only glob(CORPUS_DIR/*.txt) and the submitted config.json. It writes the tokeniz

train.sh runs src/train.py, which reads only glob(CORPUS_DIR/*.txt) and the submitted config.json. It writes the tokenizer, configs, model_*.safetensors, logs and manifest only to OUTPUT_DIR; the compile caches go to TMPDIR and are not data. No network calls or pretrained loads appear in train.py, tok.py or model.py. The public validation set is not read by training. DATA.md declares the BabyLM training files with their sha256 hashes.

train.shsrc/train.pysrc/tok.pyDATA.mdlogs/data_provenance.txt

✓serving uses only replay outputsserve.sh runs src/serve.py. The Scorer loads only WEIGHTS_DIR/config.json, tokenizer.json, model*.safetensors and config

serve.sh runs src/serve.py. The Scorer loads only WEIGHTS_DIR/config.json, tokenizer.json, model*.safetensors and config_*.json, and reads train_log.txt for a printed summary. These are all train.py outputs. It also reads the submitted serve_config.json. Serving does not read the corpus, any other model, a cache or the network.

serve.shsrc/serve.pyserve_config.jsonsrc/tok.py

✓causal normalized scoringserve_config.json sets dyn=false, so the plain path is used. Both /score and /dist use the same _logprobs on the same mo

serve_config.json sets dyn=false, so the plain path is used. Both /score and /dist use the same _logprobs on the same models, with a causal SDPA (is_causal=True), and the windows from model.plan and block_context depend only on earlier tokens. Ensemble members are averaged in probability space with logsumexp minus log K. /dist returns log_softmax over the full vocabulary and renormalizes with logsumexp; BOS gets logit -1e4 but stays inside the normalized distribution. There is no request cache, no lookup of earlier requests and no special handling for audit requests. The self-check reported score/dist agreement of 0.0003 nats (trajectory#191).

src/serve.pysrc/model.pyserve_config.jsontrajectory#191

✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC/OPENAI/other harness API keys or base URLs in the participant's comma

Searching the trajectory found no use of ANTHROPIC/OPENAI/other harness API keys or base URLs in the participant's commands, no other harness CLI sessions, and no model API calls. External services used were Spin compute and Spin transfer credentials (the participant's own allowance), used to presign a URL for the corpus upload (trajectory#49, trajectory#50). HF_TOKEN was only mentioned in the task prompt (trajectory#1).

trajectory#1trajectory#49trajectory#50trajectory#197

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
passed2h 48m into the runtook 3m 17s
startup 4.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 123 s, writing 8 files (436767805 bytes) to OUTPUT_DIR. End of its log: .txt: 12159704 bytes, 3532012 tokens, 386046 lines [ 11.0s] tokenized simple_wiki.train.txt: 8861946 bytes, 2631287 tokens, 58709 lines [ 11.1s] tokenized switchboard.train.txt: 122528 bytes, 37948 tokens, 2892 lines [ 11.1s] train tokens 15308807, dev docs 0 (0 bytes) [ 12.5s] [m0] seed 1234: params 218.1M (non-embedding 201.3M) [ 17.6s] [m0] micro_batch 32 x seq 2048, accum 2 -> 131072 tokens/step; 7481 windows/epoch; total steps 350 [ 104.4s] [m0] step 1 ep 0.01 loss 9.0108 ema 9.0108 lr 0.010 p 0.000 tok/s 1511 [ 104.9s] [m0] step 2 ep 0.02 loss 8.9939 ema 9.0091 lr 0.010 p 0.003 tok/s 271686 [ 105.4s] [m0] step 3 ep 0.03 loss 8.9796 ema 9.0061 lr 0.010 p 0.006 tok/s 260555 [ 105.4s] [m0] compiled; 10s until deadline [ 115.5s] [m0] training done: 23 steps, 0.20 epochs [ 115.8s] member 0 saved after 104s; -0s left [ 115.9s] stopping ensemble at 1 members: -0s left, last member took 104s [ 115.9s] done: 1 member(s); total wall 120.5s since REPLAY_START
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.6 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1443}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 975}
- ✓POST /dist · {"prefix_tokens": 721, "logprobs": 8192, "logsumexp": 0.0, "score_minus_dist": -1.1123114163069658e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.8 s and 86 of 516 /dist and truncation requests in 2.7 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.24 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 589 shared positions of 2 truncated texts
- ✓healthy after requests
2
passed2h 53m into the runtook 3m 35s
startup 4.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 154 s, writing 8 files (436768666 bytes) to OUTPUT_DIR. End of its log: poch; total steps 350 [ 85.5s] [m0] step 1 ep 0.01 loss 9.0108 ema 9.0108 lr 0.010 p 0.000 tok/s 1938 [ 86.0s] [m0] step 2 ep 0.02 loss 8.9939 ema 9.0091 lr 0.010 p 0.003 tok/s 282577 [ 86.5s] [m0] step 3 ep 0.03 loss 8.9796 ema 9.0061 lr 0.010 p 0.006 tok/s 272595 [ 86.5s] [m0] compiled; 61s until deadline [ 97.3s] [m0] step 25 ep 0.21 loss 4.9002 ema 8.5956 lr 1.000 p 0.171 tok/s 265848 [ 109.7s] [m0] step 50 ep 0.43 loss 4.4219 ema 8.1782 lr 1.000 p 0.375 tok/s 264993 [ 122.0s] [m0] step 75 ep 0.64 loss 4.0695 ema 7.7673 lr 0.844 p 0.578 tok/s 266400 [ 134.3s] [m0] step 100 ep 0.86 loss 3.8246 ema 7.3730 lr 0.438 p 0.781 tok/s 266705 [ 146.6s] [m0] step 125 ep 1.07 loss 3.3175 ema 6.9675 lr 0.032 p 0.984 tok/s 265988 [ 147.1s] [m0] training done: 126 steps, 1.08 epochs [ 147.5s] member 0 saved after 134s; -0s left [ 147.5s] stopping ensemble at 1 members: -0s left, last member took 134s [ 147.5s] done: 1 member(s); total wall 150.5s since REPLAY_START
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.5 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1443}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 975}
- ✓POST /dist · {"prefix_tokens": 721, "logprobs": 8192, "logsumexp": 0.0, "score_minus_dist": -5.096149791050664e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 3.0 s and 86 of 516 /dist and truncation requests in 1.4 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.18 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2450 shared positions of 2 truncated texts
- ✓healthy after requests
3
passed3h 13m into the runtook 3m 36s
startup 4.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 153 s, writing 8 files (436768804 bytes) to OUTPUT_DIR. End of its log: poch; total steps 350 [ 74.7s] [m0] step 1 ep 0.01 loss 9.0108 ema 9.0108 lr 0.010 p 0.000 tok/s 2310 [ 75.2s] [m0] step 2 ep 0.02 loss 8.9939 ema 9.0091 lr 0.010 p 0.003 tok/s 282165 [ 75.7s] [m0] step 3 ep 0.03 loss 8.9796 ema 9.0061 lr 0.010 p 0.006 tok/s 268034 [ 75.7s] [m0] compiled; 72s until deadline [ 86.6s] [m0] step 25 ep 0.21 loss 4.9064 ema 8.5962 lr 1.000 p 0.146 tok/s 264081 [ 99.0s] [m0] step 50 ep 0.43 loss 4.4387 ema 8.1804 lr 1.000 p 0.319 tok/s 263905 [ 111.4s] [m0] step 75 ep 0.64 loss 4.1097 ema 7.7733 lr 1.000 p 0.492 tok/s 264893 [ 123.7s] [m0] step 100 ep 0.86 loss 3.8641 ema 7.3824 lr 0.670 p 0.665 tok/s 265457 [ 136.0s] [m0] step 125 ep 1.07 loss 3.2884 ema 6.9730 lr 0.326 p 0.837 tok/s 266108 [ 147.4s] [m0] training done: 148 steps, 1.27 epochs [ 147.6s] member 0 saved after 135s; -0s left [ 148.1s] stopping ensemble at 1 members: -1s left, last member took 135s [ 148.1s] done: 1 member(s); total wall 150.9s since REPLAY_START
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.3 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1443}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 975}
- ✓POST /dist · {"prefix_tokens": 721, "logprobs": 8192, "logsumexp": 2.220446049250313e-16, "score_minus_dist": -1.0238319969602117e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.4 s and 86 of 516 /dist and truncation requests in 1.9 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.20 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2318 shared positions of 2 truncated texts
- ✓healthy after requests

Data-efficient pretraining

Score

Verifier details