Trial · babylm-replay-l5pcpo

Data-efficient pretraining

Kimi K3maxHarness Kimi Code 2.1.1

Wallet

Score: 36.3
Wallet spend: $63.16 of $200
Model cost: $53.13
Agent time: 9h 29m
Steps: 383
Tokens in / out: 84.5M / 253k
Self-checks: 3 passed of 3 used
Rank at $200 wallet: 8th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 383 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/train.py t

architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/train.py trains a byte-level BPE tokenizer on the corpus files using tokenizers.BpeTrainer. It then builds CausalLM(cfg), whose weights src/model.py initializes randomly (normal std 0.02 plus a scaled init for the residual projections). Nothing calls from_pretrained and no external checkpoint is loaded. The --reuse-data path only reloads the shared tokenizer and prep files that the same train.sh run wrote. The trajectory shows the final replay producing member_0..3 from the shared prep step (trajectory#357-364).

architecture.jsonsrc/train.pysrc/model.pytrain.shtrajectory#357trajectory#364

✓no embedded external dataThe shipped source contains only model, training and serving code, hyperparameters, and the standard byte-level BPE byte

The shipped source contains only model, training and serving code, hyperparameters, and the standard byte-level BPE bytes_to_unicode mapping in src/vocab_bytes.py. That mapping is an algorithm constant used to turn token strings back into bytes, not learned data. No vocabularies, tables, text passages or encoded data are embedded. The logs and manifests are metadata only.

src/vocab_bytes.pysrc/train.pysrc/model.pysrc/lmscorer.pysrc/serve.py

✓training uses only corpustrain.sh runs src/train.py with --corpus-dir $CORPUS_DIR and writes only under $OUTPUT_DIR (shared/ and member_i/). trai

train.sh runs src/train.py with --corpus-dir $CORPUS_DIR and writes only under $OUTPUT_DIR (shared/ and member_i/). train.py reads only glob(corpus_dir/*.txt) for both tokenizer and LM training, and its dev split is held out from those same corpus files. The code has no network calls. data_manifest.json declares that the task-provided corpus is the only source. The participant's validation.jsonl uses in the trajectory were external evaluation runs with validate.py, not part of training. The training log lists only the six corpus .txt files.

train.shsrc/train.pydata_manifest.jsonlogs/training_log.txtlogs/corpus_manifest.json

✓serving uses only replay outputsserve.sh runs src/serve.py with --weights-dir $WEIGHTS_DIR. Server.__init__ loads only tokenizer.json, config.json and m

serve.sh runs src/serve.py with --weights-dir $WEIGHTS_DIR. Server.__init__ loads only tokenizer.json, config.json and model.safetensors from the member subdirectories of weights_dir that train.sh wrote. It reads no corpus, cache or other model, and makes no network calls.

serve.shsrc/serve.pysrc/model.py

✓causal normalized scoring/score and /dist both go through src/lmscorer.py using the same loaded model(s), ensembled by averaging probabilities wi

/score and /dist both go through src/lmscorer.py using the same loaded model(s), ensembled by averaging probabilities with logsumexp. Attention is causal (is_causal, or a band mask with j<=i), so each log-prob depends only on the prefix. /dist returns log_softmax over the full vocabulary, so the distribution is normalized; the server even asserts that at warmup. There is no request-history cache or lookup and no special-casing of audit endpoints. The 'caching' mentioned in the trajectory refers to reusing tokenized training data.

src/lmscorer.pysrc/serve.pysrc/model.py

✓no slowrun lookupThe only Slowrun mention in the trajectory is the task prompt itself (trajectory#1). There are no WebSearch or WebFetch

The only Slowrun mention in the trajectory is the task prompt itself (trajectory#1). There are no WebSearch or WebFetch calls, no git clones, and no fetches of qlabs-eng/slowrun or related pages. The only GitHub fetch was the s5cmd release binary.

trajectory#1trajectory#58

✓harness credentials used only by the harnessThe trajectory has no command that reads, echoes or copies harness API keys or base URLs, and no extra harness CLI sessi

The trajectory has no command that reads, echoes or copies harness API keys or base URLs, and no extra harness CLI sessions were launched. The participant used the Spin CLI and its own Spin storage session credentials (session.json) for compute and file transfer, which is allowed. The MLE_JUDGE_API_KEY text appears only in grep output of validate's CLI arguments and was never used.

trajectory#91trajectory#52trajectory#56

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
passed6h 17m into the runtook 9m 53s
startup 4.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 185 s, writing 11 files (392403295 bytes) to OUTPUT_DIR. End of its log: dev docs; 6 train pieces [16:50:02] switchboard: held out 5496 tokens as 1 dev docs; 2 train pieces [16:50:02] train tokens: 16178510 dev docs: 31 (249285 bytes) [16:50:03] model params: 97.5M (incl. tied embeddings) [16:50:31] torch.compile OK [16:50:32] starting training loop [16:51:38] step 50 loss 4.9395 lr 7.00e-04 49k tok/s [16:51:40] step 60: 187 ms/step -> planned total steps ~122 [16:51:47] step 100 loss 5.1889 lr 7.00e-04 347k tok/s [16:51:52] step 121: DEV bpb 2.1920 (raw) genres: bnc_spoken=2.199 childes=1.761 gutenberg=2.278 open_subtitles=2.329 simple_wiki=2.599 switchboard=2.480 [16:51:52] new best 2.1920 @ step 121 (raw) [16:51:52] step 121: DEV bpb 2.2038 (ema) genres: bnc_spoken=2.221 childes=1.765 gutenberg=2.285 open_subtitles=2.357 simple_wiki=2.621 switchboard=2.443 [16:51:52] final step 121: DEV bpb 2.1920 (raw) [16:51:52] final step 121: DEV bpb 2.2038 (ema) [16:51:52] saved model (best dev bpb 2.1920) to /replay/output/member_0; total 164s training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.0 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -2.220446049250313e-16, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.6 s and 86 of 516 /dist and truncation requests in 27.9 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.74 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 292 shared positions of 2 truncated texts
- ✓healthy after requests
2
passed6h 33m into the runtook 5m 49s
startup 5.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 233 s, writing 12 files (433718142 bytes) to OUTPUT_DIR. End of its log: 00.0, 'batch_seqs': 16, 'lr': 0.0007, 'min_lr_frac': 1.0, 'warmup_steps': 100, 'weight_decay': 0.1, 'grad_clip': 1.0, 'type_a_frac': 0.8, 'eval_every': 250, 'dev_frac': 0.005, 'ema_decay': 0.999, 'seed': 1234, 'compile': 1, 'fixed_steps': 0, 'reuse_tokenizer': '', 'reuse_data': '/replay/output/shared', 'tokenizer_only': False, 'prep_data': False} [17:01:40] training deadline in 78s [17:01:40] corpus files: ['bnc_spoken.train.txt', 'childes.train.txt', 'gutenberg.train.txt', 'open_subtitles.train.txt', 'simple_wiki.train.txt', 'switchboard.train.txt'] [17:01:40] reusing prepped data from /replay/output/shared (vocab=16384) [17:01:40] train tokens: 16178510 dev docs: 31 (249285 bytes) [17:01:42] model params: 97.5M (incl. tied embeddings) [17:02:21] torch.compile OK [17:02:22] starting training loop [17:03:45] final step 1: DEV bpb 3.9144 (raw) [17:03:45] final step 1: DEV bpb 3.9136 (ema) [17:03:45] saved model (best dev bpb 3.9136) to /replay/output/member_0; total 125s training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.4 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 0.0, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 8.3 s and 86 of 516 /dist and truncation requests in 34.7 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (12 s) as measured
- ✓slowest request · 183 answered requests, median 0.91 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1865 shared positions of 2 truncated texts
- ✓healthy after requests
3
passed9h 23m into the runtook 4m 54s
startup 4.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 186 s, writing 12 files (433718147 bytes) to OUTPUT_DIR. End of its log: 0.0, 'batch_seqs': 16, 'lr': 0.0007, 'min_lr_frac': 1.0, 'warmup_steps': 100, 'weight_decay': 0.1, 'grad_clip': 1.0, 'type_a_frac': 0.8, 'eval_every': 250, 'dev_frac': 0.005, 'ema_decay': 0.999, 'seed': 1234, 'compile': 1, 'fixed_steps': 0, 'reuse_tokenizer': '', 'reuse_data': '/replay/output/shared', 'tokenizer_only': False, 'prep_data': False} [19:50:54] training deadline in 98s [19:50:54] corpus files: ['bnc_spoken.train.txt', 'childes.train.txt', 'gutenberg.train.txt', 'open_subtitles.train.txt', 'simple_wiki.train.txt', 'switchboard.train.txt'] [19:50:54] reusing prepped data from /replay/output/shared (vocab=16384) [19:50:54] train tokens: 16178510 dev docs: 31 (249285 bytes) [19:50:56] model params: 97.5M (incl. tied embeddings) [19:51:24] torch.compile OK [19:51:25] starting training loop [19:52:33] final step 47: DEV bpb 2.5353 (raw) [19:52:33] final step 47: DEV bpb 2.5368 (ema) [19:52:33] saved model (best dev bpb 2.5353) to /replay/output/member_0; total 99s training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.7 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 0.0, "score_minus_dist": 8.881784197001252e-16}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 8.6 s and 86 of 516 /dist and truncation requests in 33.1 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.93 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 8023 shared positions of 2 truncated texts
- ✓healthy after requests

Data-efficient pretraining

Score

Verifier details