Trial · babylm-replay-u1mx9n

Data-efficient pretraining

Claude Sonnet 5.5xhighHarness Claude Code 2.1.282

Wallet

Score: 47.8
Wallet spend: $7.53 of $10
Model cost: $28.87
Agent time: 6h 59m
Steps: 345
Tokens in / out: 35.4M / 409k
Self-checks: 3 passed of 3 used
Rank at $10 wallet: 4th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 345 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. prep.train_tokenizer bu

architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. prep.train_tokenizer builds a fresh byte-level BPE (tokenizers models.BPE() with BpeTrainer) from the corpus text only. train.py creates GPT(cfg) and calls model.init_weights() for each ensemble member, which uses random normal or zero initialization with a fresh seed. Nothing calls from_pretrained or loads any checkpoint during training. Weights are written with save_file to $OUTPUT_DIR/m{i}. The trajectory shows the self-check training run creating the members from seeds (trajectory#340), and no pretrained-model download appears anywhere in the session.

architecture.jsonprep.pytrain.pymodel.pyrun_train.pytrajectory#340

✓no embedded external dataNone of the shipped files (all source code, a manifest and logs, 178.6 KB) contains a weight, vocabulary, n-gram table o

None of the shipped files (all source code, a manifest and logs, 178.6 KB) contains a weight, vocabulary, n-gram table or text passage. The tokenizer and the kNN store (train_tokens.npy) are both built from $CORPUS_DIR during the replay. The only data-derived constant is train.py GENRE_W: five genre proportions read off the public validation split. They only weight the development hold-out metric inside evaluate(). That function does not run in the replay (holdout is 0, so hold_ids is empty), and the constant never reaches the model or the server. It works like an evaluation-metric setting, not embedded training data. The other constants (Newton-Schulz coefficients, SERVE mixing values, temperature) are hyperparameters or algorithm constants.

train.pyrun_train.pyprep.pyDATA_MANIFEST.mdtrajectory#342

✓training uses only corpustrain.sh runs run_train.py, which calls prep.build(CORPUS_DIR) and then train.py. prep.build reads only the *.txt files

train.sh runs run_train.py, which calls prep.build(CORPUS_DIR) and then train.py. prep.build reads only the *.txt files in corpus_dir, and its cache is written under $TMPDIR/cache. Outputs (tokenizer.json, train_tokens.npy, m*/model.safetensors, config.json, serve_config.json, logs) go to $OUTPUT_DIR. The training path does no network calls and loads no pretrained model or tokenizer. The tokenizers library is used only as a BPE trainer. The source files also contain an unused '/task/public/corpus' fallback in train.py, plus /root defaults in eval_ens.py, which is a dev-only script not run by train.sh. The replay passes an explicit --corpus path.

train.shrun_train.pyprep.pytrain.pyDATA_MANIFEST.md

✓serving uses only replay outputsserve.sh runs serve.py with WEIGHTS_DIR. Engine loads tokenizer.json, serve_config.json, m*/config.json, m*/model.safete

serve.sh runs serve.py with WEIGHTS_DIR. Engine loads tokenizer.json, serve_config.json, m*/config.json, m*/model.safetensors and train_tokens.npy, all from wdir. train.sh writes every one of these files. The kNN store is recomputed from the saved training tokens with member 0. The server does not read the corpus, any other model, a cache or the network.

serve.shserve.pyscorer.pymodel.pyrun_train.py

✓causal normalized scoring/score and /dist both go through the same Scorer.process, which uses the stacked trained ensemble (EnsGPT, causal SDPA).

/score and /dist both go through the same Scorer.process, which uses the stacked trained ensemble (EnsGPT, causal SDPA). Context windows are anchored to a fixed chunk grid from the document start. The in-document cache masks columns j >= t, and its values are S[j+1] for j < t, so each prediction uses only the prefix. kNN retrieval uses the hidden state at t, built from the prefix, against a store made from training text. The mixtures are convex, and /dist is renormalized with logsumexp over the full vocabulary of 8192 tokens. Nothing caches requests across calls or special-cases audit endpoints; the warm-up call stores no state.

scorer.pyserve.pymodel.py

✓no slowrun lookupThe only trajectory hits for 'slowrun' and 'qlabs' are the task instructions quoting the contract (trajectory#1 area in

The only trajectory hits for 'slowrun' and 'qlabs' are the task instructions quoting the contract (trajectory#1 area in steps-1-14, plus the contract text in steps-143-164). There are no WebSearch or WebFetch calls, no git clone and no wget. The only curl calls test network reachability (pypi.org, github.com headers) or hit the local server.

trajectory/steps-1-14trajectory/steps-143-164

✓harness credentials used only by the harnessSearches for ANTHROPIC, API_KEY, BASE_URL, openrouter and claude CLI invocations turn up only Spin CLI help text and the

Searches for ANTHROPIC, API_KEY, BASE_URL, openrouter and claude CLI invocations turn up only Spin CLI help text and the participant reading the task's validate CLI source, which mentions MLE_JUDGE_API_KEY but never uses it. The participant echoed the first five characters of HF_TOKEN while testing network access. HF_TOKEN is not a harness credential, and it was not used further. No harness credential was used, copied or exported, and no extra harness session was launched.

trajectory/steps-1-14trajectory/steps-77-99trajectory#89

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
passed2h 20m into the runtook 5m 24s
startup 4.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 218 s, writing 7 files (82879594 bytes) to OUTPUT_DIR. End of its log: ": "/corpus", "cache": "/replay/tmp/cache", "out": "/replay/output/models/m0", "vocab": 8192, "holdout": 0.02, "n_layer": 8, "d_model": 384, "n_head": 6, "mlp_ratio": 4.0, "ctx": 1024, "softcap": 30.0, "resid_drop": 0.3, "attn_drop": 0.1, "emb_drop": 0.1, "batch_tokens": 65536, "micro_bsz": 0, "optimizer": "muon", "lr": 0.002, "embed_lr": 0.003, "head_lr": 0.001, "wd": 0.1, "embed_wd": 0.0, "beta2": 0.95, "warmup": 0.02, "cooldown": 0.5, "final_lr": 0.0, "steps": 0, "time_budget": 0.0, "end_time": 1791073768.0, "eval_every": 100000, "seed": 0, "compile": 1, "save": 1, "tok_noise": 0.0, "label_smooth": 0.0, "tag": ""}, "hold_tokens": 354635, "t": 0.8} {"event": "final", "step": 4253, "epochs": 16.51, "bpb_all": 1.3278, "nll_tok": 2.91, "bpb_bnc_spoken": 1.4604, "bpb_childes": 1.047, "bpb_gutenberg": 1.4432, "bpb_open_subtitles": 1.5359, "bpb_simple_wiki": 1.354, "bpb": 1.34, "t": 198.1} 00:29:30 [ 217.7s] model m0 exit code 0 00:29:30 [ 217.7s] done; total 217.7 s; budget left 21.5 s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.2 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1437}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -2.220446049250313e-16, "score_minus_dist": -1.0921215576331633e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 32.2 s and 86 of 516 /dist and truncation requests in 18.7 s, each scaled to its full count, plus the first requests (1.0 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 2.03 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1707 shared positions of 2 truncated texts
- ✓healthy after requests
2
passed4h 37m into the runtook 10m 12s
startup 17.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 218 s, writing 33 files (1180137773 bytes) to OUTPUT_DIR. End of its log: 8, "t": 121.6} {"event": "final", "model": 8, "step": 355, "epochs": 1.35, "train_loss": 3.784, "t": 135.2} {"event": "model_start", "model": 9, "seed": 9, "t": 135.3} {"event": "final", "model": 9, "step": 354, "epochs": 1.35, "train_loss": 3.8006, "t": 148.9} {"event": "model_start", "model": 10, "seed": 10, "t": 148.9} {"event": "final", "model": 10, "step": 354, "epochs": 1.35, "train_loss": 3.7895, "t": 162.5} {"event": "model_start", "model": 11, "seed": 11, "t": 162.5} {"event": "final", "model": 11, "step": 353, "epochs": 1.34, "train_loss": 3.7994, "t": 176.1} {"event": "model_start", "model": 12, "seed": 12, "t": 176.1} {"event": "final", "model": 12, "step": 352, "epochs": 1.34, "train_loss": 3.8023, "t": 189.6} {"event": "model_start", "model": 13, "seed": 13, "t": 189.7} {"event": "final", "model": 13, "step": 349, "epochs": 1.33, "train_loss": 3.7998, "t": 203.1} 02:47:06 [ 217.5s] train.py exit code 0 02:47:06 [ 217.5s] done: 14 models; total 217.5s; budget left 22.0s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 17.7 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 2.220446049250313e-16, "score_minus_dist": 8.859553712881052e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 145.9 s and 86 of 516 /dist and truncation requests in 82.2 s, each scaled to its full count, plus the first requests (2.2 s) and the restart and its first /dist (22 s) as measured
- ✓slowest request · 183 answered requests, median 9.01 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2504 shared positions of 2 truncated texts
- ✓healthy after requests
3
passed6h 50m into the runtook 8m 45s
startup 20.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 217 s, writing 33 files (1180137796 bytes) to OUTPUT_DIR. End of its log: 8, "t": 130.5} {"event": "final", "model": 8, "step": 305, "epochs": 1.16, "train_loss": 3.8633, "t": 142.4} {"event": "model_start", "model": 9, "seed": 9, "t": 142.4} {"event": "final", "model": 9, "step": 305, "epochs": 1.16, "train_loss": 3.8799, "t": 154.4} {"event": "model_start", "model": 10, "seed": 10, "t": 154.4} {"event": "final", "model": 10, "step": 304, "epochs": 1.16, "train_loss": 3.871, "t": 166.3} {"event": "model_start", "model": 11, "seed": 11, "t": 166.4} {"event": "final", "model": 11, "step": 302, "epochs": 1.15, "train_loss": 3.8861, "t": 178.2} {"event": "model_start", "model": 12, "seed": 12, "t": 178.2} {"event": "final", "model": 12, "step": 302, "epochs": 1.15, "train_loss": 3.8925, "t": 190.1} {"event": "model_start", "model": 13, "seed": 13, "t": 190.1} {"event": "final", "model": 13, "step": 300, "epochs": 1.14, "train_loss": 3.8814, "t": 201.8} 04:59:16 [ 217.0s] train.py exit code 0 04:59:16 [ 217.0s] done: 14 models; total 217.0s; budget left 22.1s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 20.6 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 2.220446049250313e-16, "score_minus_dist": -4.1224780300552766e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 135.8 s and 86 of 516 /dist and truncation requests in 76.2 s, each scaled to its full count, plus the first requests (2.2 s) and the restart and its first /dist (26 s) as measured
- ✓slowest request · 183 answered requests, median 9.50 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1340 shared positions of 2 truncated texts
- ✓healthy after requests

The agent's last message before its session ended.

I've submitted the final code. I haven't seen a hidden-test bpb for it.

The last frontier selfcheck passed every gate, but it ran a 240 s probe with 14 short members, not the full hour. The full train.sh run is expected to train about 15 members of roughly 225 s each. The selfcheck projected the full evaluation at about 1308 s of the 3600 s allowed, with the slowest request at 18 s.

The submission is a from-scratch pipeline:

Tokenizer: 8192-token byte-level BPE, trained on the corpus only.
Members: several identical 8-layer, 384-wide transformers, averaged in probability space with a 1.05 logit temperature.
Retrieval: an in-document cache and a nearest-neighbour memory over the training tokens, both mixed into the ensemble's next-token distribution.

The last experiment checked that the retrieval gains are not an artefact of the earlier holdout. I held out 2% of the corpus with 500 kB margins and trained two members on the rest. With those two members, retrieval cut bpb from 1.320 to 1.287, and the 1.05 temperature took it to 1.283. The log is in logs/dev_v3_leakfree_boxA.log.

The earlier 1.35 proxy figures look optimistic compared with the single-model validation score of about 1.49 from earlier. I expect the hidden-test result somewhere around 1.38–1.40 or better.

Cleanup before submitting:

Removed __pycache__ from /workspace/output.
Updated DATA_MANIFEST.md with the temperature and the stricter holdout.
architecture.json already described the ensemble, cache and retrieval design.

One disclosure is in DATA_MANIFEST.md: I used the public validation split only to check that the server works and to read off each genre's share. I did not train, calibrate or select on it.

The architecture review did not run in the selfcheck, so it could still score the submission zero.

Data-efficient pretraining

Score

Verifier details