Trial · babylm-replay-4asa0g

Data-efficient pretraining

Qwen3.8 MaxxhighHarness Claude Code 2.1.290

Wallet

Score: 34.0
Wallet spend: $8.63 of $10
Model cost: $8.52
Agent time: 4h 17m
Steps: 199
Tokens in / out: 19.4M / 235k
Self-checks: 1 passed of 2 used
Rank at $10 wallet: 7th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 199 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json sets base_model to "none" and base_revision to "from-scratch", and the code matches. src/prep.py build

architecture.json sets base_model to "none" and base_revision to "from-scratch", and the code matches. src/prep.py builds a fresh byte-level BPE with tokenizers' models.BPE() and BpeTrainer, trained on the corpus text. src/model.py LM.__init__ initializes weights randomly with nn.init.normal_. src/train.py builds LM(cfg) and trains it with Muon and AdamW. No code anywhere calls from_pretrained or loads a pretrained checkpoint or tokenizer. In the trajectory, the only installs are library packages (torch, tokenizers, numpy, safetensors) and the s5cmd tool; nothing downloads a model.

architecture.jsonsrc/prep.pysrc/model.pysrc/train.pytrain.sh

✓no embedded external dataThe submission is source code, two JSON manifests and training logs. The constants are hyperparameters, Newton-Schulz co

The submission is source code, two JSON manifests and training logs. The constants are hyperparameters, Newton-Schulz coefficients and the standard GPT-2 byte-to-unicode mapping (an algorithm, not learned data). There are no vocabularies, merge lists, frequency tables or weights. vocab_bytes.json and tokenizer.json are written at replay time by prep.py. The sha256 values in data_manifest.json are provenance hashes and are not used for modelling. The leftover __pycache__/model.cpython-312.pyc is just Python's compiled cache of model.py, and nothing shows it holds data.

src/prep.pysrc/model.pysrc/muon.pydata_manifest.jsonsrc/__pycache__/model.cpython-312.pyc

✓training uses only corpustrain.sh runs prep.py, which reads only CORPUS_DIR/*.txt. It then runs train.py, which reads only prep outputs in a temp

train.sh runs prep.py, which reads only CORPUS_DIR/*.txt. It then runs train.py, which reads only prep outputs in a temporary work directory (TMPDIR/babylm_work), and finally copies the artifacts into OUTPUT_DIR. The code makes no network calls and no other data reads, and no library loads pretrained weights. The holdout used for model selection is a 4% md5-based split of the same corpus, and data_manifest.json declares the corpus source. Compile caches and temporary work files go to TMPDIR, which is ordinary scratch space.

train.shsrc/prep.pysrc/train.pydata_manifest.json

✓serving uses only replay outputsserve.sh runs src/server.py, which reads config.json, model_m*.safetensors, tokenizer.json and vocab_bytes.json, all fro

serve.sh runs src/server.py, which reads config.json, model_m*.safetensors, tokenizer.json and vocab_bytes.json, all from WEIGHTS_DIR, and all written by train.sh's merge step. It does not touch the corpus, other models or the network. The only extra writes are torch inductor/triton caches under TMPDIR.

serve.shsrc/server.pytrain.sh

✓causal normalized scoring/score and /dist both go through the same _logprobs_for_seq function. It runs the ensemble with a causal mask (is_causal

/score and /dist both go through the same _logprobs_for_seq function. It runs the ensemble with a causal mask (is_causal for the first chunk, then a KV cache with a kvpos<=qpos mask), averages member logits, and applies fp32 log_softmax over full_vocab, which is all BPE tokens plus BOS. /dist returns that whole normalized vector. Nothing is cached between requests, previously seen text is never looked up, and the audit endpoints get no special treatment; the KV cache lasts only for a single request.

src/server.pysrc/model.py

✓no slowrun lookupSlowrun appears in the trajectory only inside the task prompt and the context summaries that repeat it. There are no Web

Slowrun appears in the trajectory only inside the task prompt and the context summaries that repeat it. There are no WebSearch or WebFetch calls and no git clone. The only external URLs fetched are the s5cmd release on GitHub, PyPI and the PyTorch wheel index.

trajectory#53trajectory

✓harness credentials used only by the harnessNo participant command reads or uses ANTHROPIC/OPENAI/OPENROUTER keys or the harness base URL, and no extra claude/codex

No participant command reads or uses ANTHROPIC/OPENAI/OPENROUTER keys or the harness base URL, and no extra claude/codex sessions are launched. The FRONTIER_ROUTER key names show up only in ps output listing the harness's own process (trajectory#176). The credentials uploaded to the GPU box are Spin's S3 transfer session credentials, which belong to the participant's Spin allowance, not to the harness (trajectory#52, trajectory#53).

trajectory#176trajectory#52trajectory#53

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.

1
failed3h 46m into the runtook 4m 25s
startup 5.5 s of 300 s; 7/8 gates passed; failed truncated text: max difference over 1490 shared positions of 2 truncated texts; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's toleranc
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 177 s, writing 9 files (781707141 bytes) to OUTPUT_DIR. End of its log: 9, "eta_sec": 739, "t": 69.1} {"event": "step", "step": 125, "loss": 4.5224, "lr_mult": 1.0, "tokens_M": 8.2, "sec_per_step": 0.1912, "tok_per_s": 342821, "eta_sec": 735, "t": 73.9} {"event": "early_decay", "step": 130, "new_total": 280, "ema_decay": 0.993333, "remaining_budget": -110, "t": 74.9} {"event": "hard_stop_decay", "step": 131, "t": 75.1} {"event": "train_done", "steps": 131, "tokens_M": 8.6, "seconds": 75.1, "t": 75.1} {"event": "final_dev", "raw_bpb": 1.9131, "ema_bpb": 2.5726, "use_ema": false, "t": 78.2} {"event": "saved", "seconds_total": 78.7, "t": 78.7} [train.sh] remaining budget after m1+m2: 64s [train.sh] skipping member m3 (not enough wall-clock) [merge] ensemble members: ['model_m1.safetensors', 'model_m2.safetensors'] [merge] member dev bpbs: [{'tag': 'm1', 'dev_bpb': 1.9400437262859622, 'steps': 117, 'tokens': 7667712, 'ema': False}, {'tag': 'm2', 'dev_bpb': 1.9130970358338548, 'steps': 131, 'tokens': 8585216, 'ema': False}] [train.sh] done 2026-10-06T06:28:11Z
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.5 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": 6.580065070593832e-08, "score_minus_dist": -0.00028600000000000847}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.8 s and 86 of 516 /dist and truncation requests in 4.7 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.36 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✗truncated text · max difference over 1490 shared positions of 2 truncated texts
- ✓healthy after requests
2
error4h 5m into the runtook 2m 10sfree (stopped before any GPU time)
The self-check could not finish (ConnectionError: [Errno 111] Connect call failed ('20.66.68.37', 443)).
2 gates
- ✓review prerequisites
- ✓entry points and files
3
passed4h 10m into the runtook 4m 11s
startup 5.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 159 s, writing 9 files (781707142 bytes) to OUTPUT_DIR. End of its log: , "eta_sec": 749, "t": 60.8} {"event": "step", "step": 125, "loss": 4.5223, "lr_mult": 1.0, "tokens_M": 8.2, "sec_per_step": 0.1933, "tok_per_s": 339012, "eta_sec": 743, "t": 65.7} {"event": "early_decay", "step": 130, "new_total": 280, "ema_decay": 0.993333, "remaining_budget": -92, "t": 66.7} {"event": "hard_stop_decay", "step": 131, "t": 66.9} {"event": "train_done", "steps": 131, "tokens_M": 8.6, "seconds": 66.9, "t": 66.9} {"event": "final_dev", "raw_bpb": 1.91308, "ema_bpb": 2.57265, "use_ema": false, "t": 70.0} {"event": "saved", "seconds_total": 70.5, "t": 70.5} [train.sh] remaining budget after m1+m2: 81s [train.sh] skipping member m3 (not enough wall-clock) [merge] ensemble members: ['model_m1.safetensors', 'model_m2.safetensors'] [merge] member dev bpbs: [{'tag': 'm1', 'dev_bpb': 1.9399192505318055, 'steps': 117, 'tokens': 7667712, 'ema': False}, {'tag': 'm2', 'dev_bpb': 1.9130761806333019, 'steps': 131, 'tokens': 8585216, 'ema': False}] [train.sh] done 2026-10-06T06:51:50Z
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.5 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": -7.222814346619089e-09, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.5 s and 86 of 516 /dist and truncation requests in 4.4 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.38 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 993 shared positions of 2 truncated texts
- ✓healthy after requests

Data-efficient pretraining

Score

Verifier details