Trial · babylm-replay-3ya3rs

Data-efficient pretraining

Kimi K3maxHarness Kimi Code 2.1.1

Wallet

Score: 13.7
Wallet spend: $38.39 of $50
Model cost: $36.30
Agent time: 10h 17m
Steps: 420
Tokens in / out: 100.5M / 292k
Self-checks: 2 passed of 3 used
Rank at $50 wallet: 9th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 420 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In src/model.py, GP

architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In src/model.py, GPT builds its weights with nn.init.normal_ and zeros_ (the embeddings are tied by default). src/train.py constructs GPT(cfg) under a seed and never loads a checkpoint before training. The tokenizer is an empty tokenizers BPE model that train_tokenizer() in src/tokenizer_util.py trains on the corpus files. No from_pretrained calls, hub downloads or library-provided weights appear anywhere, and the trajectory shows no model downloads.

architecture.jsonsrc/model.pysrc/train.pysrc/tokenizer_util.py

✓no embedded external dataThe shipped files contain only code, hyperparameters and evidence logs or manifests. The vocabulary and merges are learn

The shipped files contain only code, hyperparameters and evidence logs or manifests. The vocabulary and merges are learned at train time and not shipped. The GPT-2 bytes-to-unicode mapping in tokenizer_util.py is a byte-encoding algorithm constant, not a data table. data_manifest.json holds only checksums of the corpus files. No embedded weights, word lists, n-gram tables or encoded data were found.

src/tokenizer_util.pysrc/train.pysrc/model.pyevidence/data_manifest.json

✓training uses only corpustrain.sh runs src/train.py, which lists the *.txt files under $CORPUS_DIR, trains the tokenizer on them, encodes them, a

train.sh runs src/train.py, which lists the *.txt files under $CORPUS_DIR, trains the tokenizer on them, encodes them, and holds out part of the corpus for validation. It writes tokenizer.json, model_*.safetensors, serve_config.json, data_manifest.json, training_summary.json and train_log.txt only to $OUTPUT_DIR. The code makes no network calls and reads no other data paths. The corpus copied to the remote machine came from /task/public/corpus (trajectory#64 region, tar of /task/public corpus). The data_manifest checksums and byte sizes match the corpus files.

train.shsrc/train.pysrc/tokenizer_util.pyevidence/data_manifest.jsontrajectory/steps-55-99

✓serving uses only replay outputsserve.sh runs src/serve.py. The Scorer reads serve_config.json, tokenizer.json and the model_*.safetensors files listed

serve.sh runs src/serve.py. The Scorer reads serve_config.json, tokenizer.json and the model_*.safetensors files listed in the config, all from $WEIGHTS_DIR, which train.sh writes. It does not read the corpus, any other model or cache, or the network.

serve.shsrc/serve.pysrc/train.py

✓causal normalized scoring/score and /dist both call _token_logprobs. That function runs the ensemble through a single forward pass with causal at

/score and /dist both call _token_logprobs. That function runs the ensemble through a single forward pass with causal attention (is_causal=True in model.py), averages the softmax probabilities across models, and renormalizes over the full vocabulary with logsumexp. /score takes the log-prob of token i from position i of a BOS-prefixed input, and /dist takes the last position. There is no cache of earlier requests and no lookup of previously seen text. The only cache is the RoPE table keyed by sequence length, which holds no request data. Neither audit endpoint is special-cased.

src/serve.pysrc/model.py

✓no slowrun lookupSearching the trajectory for 'slowrun', 'Slowrun', 'github', 'WebSearch', 'git clone' and 'wget' finds only the task pro

Searching the trajectory for 'slowrun', 'Slowrun', 'github', 'WebSearch', 'git clone' and 'wget' finds only the task prompt and CONTRACT.md text that name Slowrun (trajectory#1, trajectory#2). There are no web searches, fetches or clones aimed at it.

trajectory#1trajectory#2trajectory/steps-1-19

✓harness credentials used only by the harnessThe trajectory shows no use of ANTHROPIC, OPENAI or OPENROUTER keys or base URLs, and no launches of extra claude, codex

The trajectory shows no use of ANTHROPIC, OPENAI or OPENROUTER keys or base URLs, and no launches of extra claude, codex or similar CLI sessions. The only API_KEY and BASE_URL matches are the Spin CLI help text. Remote compute went through the Spin CLI, which is allowed.

trajectory/steps-1-19trajectory/steps-55-99

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
failed3h 10m into the runtook 17m 9s
startup 2.5 s of 300 s; 1/1 gates passed; failed POST /score: RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": "internal error"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "internal error"}' | RuntimeError: Submission retur
7 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 NVL after 21 s, writing 5 files (2297891 bytes) to OUTPUT_DIR. End of its log: in 05:57:09] bnc_spoken.train.txt: 44 docs, held out 2 docs / 3000 lines for val [train 05:57:09] childes.train.txt: 1566 docs, held out 18 docs / 8034 lines for val [train 05:57:09] gutenberg.train.txt: 60 docs, held out 2 docs / 1647 lines for val [train 05:57:09] open_subtitles.train.txt: 155 docs, held out 2 docs / 5000 lines for val [train 05:57:09] simple_wiki.train.txt: 14800 docs, held out 187 docs / 709 lines for val [train 05:57:09] switchboard.train.txt: 5 docs, held out 2 docs / 792 lines for val [train 05:57:10] train blocks 3038 (15.6M tok), val: 176 docs, 970976 bytes (bnc_spoken:16, childes:48, gutenberg:48, open_subtitles:32, simple_wiki:16, switchboard:16) [train 05:57:11] model config {'vocab_size': 32768, 'd_model': 768, 'n_layer': 12, 'n_head': 12, 'd_ff': 2048, 'dropout': 0.1, 'rope_theta': 10000.0, 'bos_id': 32767, 'tie_emb': 0} params 135.3M [train 05:57:11] not enough time left for another model; stopping [train 05:57:11] ALL DONE in 17s; models: 0; val bpb []
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.5 s
- ✓GET /vocab · {"tokens": 32768, "special": 1}
- ✗POST /score · RuntimeError: http_status: failed 3 attempts: RuntimeError: Submission returned HTTP 500: b'{"error": "internal error"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "internal error"}' | RuntimeError: Submission returned HTTP 500: b'{"error": "internal error"}'
2
passed7h 55m into the runtook 5m 10s
startup 4.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 213 s, writing 6 files (441611728 bytes) to OUTPUT_DIR. End of its log: le_wiki:16, switchboard:16) [train 10:26:51] model config {'vocab_size': 16384, 'd_model': 768, 'n_layer': 12, 'n_head': 12, 'd_ff': 2048, 'dropout': 0.2, 'rope_theta': 10000.0, 'bos_id': 16383, 'tie_emb': 1} params 97.5M [train 10:26:51] model 0: elapsed 18s, budget for this model 60s, initial planned steps 200 [train 10:27:41] [m0] torch.compile ok [train 10:27:41] [m0] blocks 3151, steps/epoch 196, planned 200 [train 10:28:23] [m0] calibrated 1.425s/step -> planned steps 200 -> 80 [train 10:28:29] [m0] step 50/80 epoch~0.26 loss 7.5857 lr 3.00e-04 0.959s/step [train 10:29:59] [m0] eval step 80: val bpb(w) raw 2.1208 ema 3.6273 ema2 3.9964 [train 10:29:59] [m0] new best: raw @ step 80, bpb(w) 2.1208 [train 10:29:59] [m0] finished 80 steps in 139s (1.734s/step); best raw @ 80 bpb(w) 2.1208 [train 10:30:00] model 0 saved to /replay/output/model_0.safetensors [train 10:30:00] not enough time left for another model; stopping [train 10:30:00] ALL DONE in 207s; models: 1; val bpb [2.1208]
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.0 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 6.033783606085308e-08, "score_minus_dist": -7.152557373046875e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.3 s and 86 of 516 /dist and truncation requests in 4.1 s, each scaled to its full count, plus the first requests (2.4 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.56 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1165 shared positions of 2 truncated texts
- ✓healthy after requests
3
passed9h 58m into the runtook 4m 51s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 217 s, writing 6 files (441611731 bytes) to OUTPUT_DIR. End of its log: le_wiki:16, switchboard:16) [train 12:29:45] model config {'vocab_size': 16384, 'd_model': 768, 'n_layer': 12, 'n_head': 12, 'd_ff': 2048, 'dropout': 0.2, 'rope_theta': 10000.0, 'bos_id': 16383, 'tie_emb': 1} params 97.5M [train 12:29:45] model 0: elapsed 17s, budget for this model 60s, initial planned steps 200 [train 12:30:37] [m0] torch.compile ok [train 12:30:37] [m0] blocks 3151, steps/epoch 196, planned 200 [train 12:31:21] [m0] calibrated 1.452s/step -> planned steps 200 -> 80 [train 12:31:26] [m0] step 50/80 epoch~0.26 loss 7.5848 lr 3.00e-04 0.975s/step [train 12:32:59] [m0] eval step 80: val bpb(w) raw 2.1133 ema 3.6274 ema2 3.9967 [train 12:32:59] [m0] new best: raw @ step 80, bpb(w) 2.1133 [train 12:32:59] [m0] finished 80 steps in 142s (1.771s/step); best raw @ 80 bpb(w) 2.1133 [train 12:32:59] model 0 saved to /replay/output/model_0.safetensors [train 12:32:59] not enough time left for another model; stopping [train 12:32:59] ALL DONE in 211s; models: 1; val bpb [2.1133]
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 8.416007535316794e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.3 s and 86 of 516 /dist and truncation requests in 2.8 s, each scaled to its full count, plus the first requests (1.7 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.51 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2951 shared positions of 2 truncated texts
- ✓healthy after requests

Data-efficient pretraining

Score

Verifier details