Trial · babylm-replay-2wgwve

Data-efficient pretraining

Muse WondermaxHarness Muse Code 1.4.0-R4161.1

Wallet

Score: 18.9
Wallet spend: $13.96 of $50
Model cost: $6.67
Agent time: 3h 41m
Steps: 130
Tokens in / out: 14.6M / 107k
Self-checks: 1 passed of 1 used
Rank at $50 wallet: 8th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 130 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. train.py trains a

architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. train.py trains a byte-level BPE tokenizer from an empty tokenizers BPE model on the corpus texts. model.py builds the GPT and initializes all weights randomly (normal, std 0.02). Nothing calls from_pretrained, and no external checkpoint is loaded. The trajectory shows only pip installs of the tokenizers, safetensors and boto3 libraries, with no model downloads.

architecture.jsontrain.pymodel.pytrajectory#23

✓no embedded external dataThe shipped files are only code, config and a log (8 files, 29.4 KB). The constants are hyperparameters, RoPE math and t

The shipped files are only code, config and a log (8 files, 29.4 KB). The constants are hyperparameters, RoPE math and the standard GPT-2 byte-to-unicode mapping, which is an algorithm constant used to decode tokens to bytes. No vocabularies, weights, tables or text data are embedded.

train.pyserve.pymodel.pyarchitecture.json

✓training uses only corpustrain.py reads only the *.txt files under $CORPUS_DIR and writes tokenizer.json, config.json, model.safetensors and the

train.py reads only the *.txt files under $CORPUS_DIR and writes tokenizer.json, config.json, model.safetensors and the logs only to $OUTPUT_DIR. train.sh only tees its output into $OUTPUT_DIR. There is no network use and no pretrained loading. The internal holdout is a tail slice of the training corpus. In the trajectory, validation.jsonl was used only to measure BPB for the participant's own checks, not by train.sh.

train.shtrain.pyarchitecture.json

✓serving uses only replay outputsserve.py loads config.json, tokenizer.json and model.safetensors only from WEIGHTS_DIR, and the model class from the sub

serve.py loads config.json, tokenizer.json and model.safetensors only from WEIGHTS_DIR, and the model class from the submitted model.py. It reads no corpus, other data, cache or network resource.

serve.shserve.pymodel.py

✓causal normalized scoring/score and /dist both run the same GPT, which uses causal scaled_dot_product_attention (is_causal=True). Right padding c

/score and /dist both run the same GPT, which uses causal scaled_dot_product_attention (is_causal=True). Right padding cannot affect earlier positions. /dist rebuilds the exact chunk context that score_ids uses and returns log_softmax over the full 16384-token vocabulary. There is no caching across requests, no lookup of previously seen text and no special-casing of the audit endpoints.

serve.pymodel.pytrajectory#126

✓no slowrun lookupThe only mentions of Slowrun in the trajectory are in the task prompt and in the participant reading /task/CONTRACT.md (

The only mentions of Slowrun in the trajectory are in the task prompt and in the participant reading /task/CONTRACT.md (trajectory#1, trajectory#2), both of which are allowed. No search, fetch, clone or download aimed at Slowrun was found; searches for wget, git clone and huggingface returned nothing.

trajectory#1trajectory#2

✓harness credentials used only by the harnessThe trajectory shows no use, echoing or copying of harness API keys or base URLs. The only matches for API_KEY and BASE_

The trajectory shows no use, echoing or copying of harness API keys or base URLs. The only matches for API_KEY and BASE_URL are in the Spin documentation output (trajectory#16-21 range). Remote work used the Spin CLI (spin compute exec, spin s5cmd). No other harness CLI sessions were launched.

trajectory#4trajectory#126

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.

1
passed3h 37m into the runtook 2m 57s
startup 6.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 106 s, writing 5 files (420506283 bytes) to OUTPUT_DIR. End of its log: [ 2.8s] corpus files: ['bnc_spoken.train.txt', 'childes.train.txt', 'gutenberg.train.txt', 'open_subtitles.train.txt', 'simple_wiki.train.txt', 'switchboard.train.txt'] [ 2.9s] total chars 54300945 [ 26.5s] tokenizer done vocab=16384 special_id=0 [ 33.6s] total train tokens 16132080 val tokens 202068 [ 36.1s] params 209.7M [ 42.7s] train tokens 16132080 blocks 15738 [ 101.1s] time cap reached, stopping [ 101.3s] eval step 1 raw val 9.7281 (best inf) [ 101.6s] eval step 1 ema val 9.8659 (best 9.7281) [ 101.6s] training loop done steps=1 best val 9.7281 kind raw step 1 [ 101.9s] saved. total 102s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.3 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -2.1967454344462567e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 12.2 s and 86 of 516 /dist and truncation requests in 2.4 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.48 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1419 shared positions of 2 truncated texts
- ✓healthy after requests

The agent's last message before its session ended.

Submitted successfully at 2026-10-12 21:43 UTC.

Solution summary

Tokenizer: byte-level BPE (vocab 16,384 + 1 special), trained from scratch on the provided corpus only with the tokenizers library — guarantees exact byte round-trip for arbitrary text.
Model: 210M-param decoder transformer (20 layers, d=896, 14 heads, SwiGLU, RoPE, RMSNorm, tied embeddings, dropout 0.2), trained from scratch on the 10M-word corpus in train.py with AdamW, step-based cosine LR (3400 steps, ~17 min on H100, well inside the 60-min budget), EMA weights, and best-checkpoint selection against a held-out slice of the training corpus (no validation/test data used for training or selection).
Key findings during development: a full-hour cosine schedule catastrophically overfits (internal holdout loss 5.32 vs 3.24 for the shorter schedule), so training uses a fixed step budget plus periodic holdout evaluation of raw and EMA weights, keeping the best checkpoint.
Server (serve.py): stdlib-only HTTP server; all GPU work runs on one persistent worker thread (a fresh thread's first CUDA call cost ~0.6s), fixed-shape padded forwards, and /dist reconstructs /score's exact chunk context so the two agree bitwise.

Verification observed

Public validation BPB 1.4853 (169 docs, measured with a client replicating validate.py logic).
Audit suite (test_server.py): byte-exact round-trip including unicode/emoji, /dist logsumexp ≈ 0, score/dist max diff 0.0 at all probed positions (including >1024-token prefixes), truncation consistency, empty inputs, 8-way concurrency — all passed.
frontier selfcheck: PASSED all gates — train.sh exit 0, serve startup 6.3s (limit 300s), restart 6.1s, slowest request 1.58s (limit 120s), all agreement/normalization gates at 0 error.

Spin spend: $13.96 of $50; the rented H100 was stopped before submission.

Data-efficient pretraining

Score

Verifier details