Trial · babylm-replay-1d854k

Data-efficient pretraining

Claude Fable 5.1xhighHarness Claude Code 2.1.282

Wallet

Score: 26.9
Wallet spend: $5.59 of $10
Model cost: $48.86
Agent time: 3h 25m
Steps: 122
Tokens in / out: 19.1M / 162k
Self-checks: 3 passed of 3 used
Rank at $10 wallet: 9th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 122 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch". The code backs this up. In train.py, trai

architecture.json declares base_model "none" and base_revision "from-scratch". The code backs this up. In train.py, train_one(k) seeds torch, builds GPT(cfg), and model.py's _init_weights fills the weights with normal(0.02) values, with zeros for the output projections and lm_head. The only checkpoint train.py loads is one it just saved itself to OUTPUT_DIR. The BPE tokenizer is trained from scratch on the corpus in bpe_tokenizer.py (BpeTrainer and train_from_iterator). No pretrained weights or tokenizer are loaded anywhere.

architecture.jsontrain.pymodel.pybpe_tokenizer.py

✓no embedded external dataThe submission ships only code, manifests and logs, with no weight files or data tables. bpe_tokenizer.py contains only

The submission ships only code, manifests and logs, with no weight files or data tables. bpe_tokenizer.py contains only the algorithmic GPT-2 bytes_to_unicode mapping, which is not external data. No embedded vocabularies, n-gram tables or external text were found.

bpe_tokenizer.pytrain.pymodel.pyDATA_MANIFEST.md

✓training uses only corpustrain.sh points CORPUS_DIR at /corpus, and train.py reads only glob(CORPUS_DIR/*.txt). The trailing-space augmentation i

train.sh points CORPUS_DIR at /corpus, and train.py reads only glob(CORPUS_DIR/*.txt). The trailing-space augmentation is generated synthetically from that same corpus text. The 1% tail holdout used for selection is also taken from the corpus. DATA_MANIFEST.md and architecture.json declare corpus-only provenance with calibration "none". train.py has no network access or downloads, and the trajectory's pip installs were software packages only.

train.shtrain.pybpe_tokenizer.pyDATA_MANIFEST.mdarchitecture.json

✓serving uses only replay outputsserve.sh sets WEIGHTS_DIR (default /replay/output) and runs serve.py. serve.py loads only config.json, tokenizer.json an

serve.sh sets WEIGHTS_DIR (default /replay/output) and runs serve.py. serve.py loads only config.json, tokenizer.json and the model_files listed in config, all from WEIGHTS_DIR, and train.py writes every one of those files into OUTPUT_DIR. Serving uses no other model or data source.

serve.shserve.pytrain.py

✓causal normalized scoringmodel.py uses SDPA with is_causal=True. In serve.py, each ensemble member's logits go through log_softmax over the full

model.py uses SDPA with is_causal=True. In serve.py, each ensemble member's logits go through log_softmax over the full vocabulary. Members are then combined with logaddexp minus log N, which is a properly normalized mixture. Special tokens are masked, and next_dist reports them as -1e4. /score and /dist use the same deterministic window start (max(0,(block+1)*STRIDE-CTX)) with BOS prepended. Nothing is cached across requests. The participant's selfcheck shows a logsumexp of about 0 and a score_minus_dist of 0, though that is only supporting evidence.

serve.pymodel.pytrajectory#116

✓no slowrun lookupThe trajectory shows no search, open or fetch of Slowrun or any derivative of it. The only mention is in the task prompt

The trajectory shows no search, open or fetch of Slowrun or any derivative of it. The only mention is in the task prompt at the start of the session.

trajectory#1trajectory#2

✓harness credentials used only by the harnessThe trajectory shows no use or copying of harness API keys or base URLs. It shows no launch of harness CLIs such as clau

The trajectory shows no use or copying of harness API keys or base URLs. It shows no launch of harness CLIs such as claude or codex, and no printenv credential dumps. The R2 transfer credentials used with s5cmd belong to the participant's own Spin allowance, which the policy permits. The frontier selfcheck calls are the sanctioned task tooling.

trajectory#28trajectory#30trajectory#116

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
passed50m 52s into the runtook 2m 40s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 93 s, writing 6 files (441624275 bytes) to OUTPUT_DIR. End of its log: tep": 20, "loss": 4.6769, "lr_mult": 0.6145, "epoch": 0.322, "tok_per_s": 396942, "progress": 0.6928, "mem_gb": 44.22, "t": 65.2} {"event": "step", "step": 30, "loss": 4.5572, "lr_mult": 0.4295, "epoch": 0.483, "tok_per_s": 396694, "progress": 0.7853, "mem_gb": 44.22, "t": 71.8} {"event": "step", "step": 40, "loss": 4.2066, "lr_mult": 0.2434, "epoch": 0.644, "tok_per_s": 393996, "progress": 0.8783, "mem_gb": 44.22, "t": 78.4} {"event": "step", "step": 50, "loss": 4.3572, "lr_mult": 0.0567, "epoch": 0.805, "tok_per_s": 393334, "progress": 0.9717, "mem_gb": 44.22, "t": 85.1} {"event": "checkpoint", "step": 53, "which": "raw", "loss": 4.2284, "save_seconds": 0.6, "t": 87.9} {"event": "final_eval", "step": 53, "loss_raw": 4.2284, "loss_ema": 4.3329, "bpb_raw": 1.8146, "bpb_ema": 1.8595, "best": {"loss": 4.228427410125732, "which": "raw", "step": 53}, "train_seconds": 72.9, "epochs": 0.854, "t": 87.9} {"event": "done", "wall": 87.9, "t": 87.9} train.py exit status 0 at 2026-10-03T22:58:49Z
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 16384, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -8.937040441026056e-09, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.1 s and 86 of 516 /dist and truncation requests in 2.4 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.24 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 7987 shared positions of 2 truncated texts
- ✓healthy after requests
2
passed3h 17m into the runtook 2m 33s
startup 4.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 67 s, writing 6 files (441623247 bytes) to OUTPUT_DIR. End of its log: 3, "attn_dropout": 0.0, "emb_dropout": 0.0, "tie_embeddings": false}, "t": 15.4} {"event": "train_start", "k": 0, "grad_accum": 4, "micro_bs": 32, "seq_len": 2048, "tokens_per_step": 262144, "slice_seconds": 16.2, "t": 19.1} {"event": "step", "k": 0, "step": 1, "loss": 9.7041, "lr_mult": 0.001, "epoch": 0.016, "tok_per_s": 0, "progress": 0.0, "mem_gb": 43.67, "t": 60.4} {"event": "checkpoint", "k": 0, "step": 1, "which": "ema", "loss": 9.7036, "bpb": 4.2713, "save_seconds": 0.6, "t": 61.7} {"event": "final_eval", "k": 0, "step": 1, "loss_raw": 9.7036, "loss_ema": 9.7036, "bpb_raw": 4.2713, "bpb_ema": 4.2713, "best": {"loss": 9.703594207763672, "which": "ema", "step": 1, "epoch": 0.016}, "train_seconds": 42.6, "epochs": 0.016, "t": 61.7} {"event": "ensemble", "n_models": 1, "holdout_loss": 9.7036, "holdout_bpb": 4.2713, "singles": [9.7036], "t": 62.2} {"event": "done", "wall": 62.2, "ensemble_holdout_bpb": 4.2713, "n_models": 1, "t": 62.2} train.py exit status 0 at 2026-10-04T01:25:14Z
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.4 s
- ✓GET /vocab · {"tokens": 16384, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.963188562825735e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.6 s and 86 of 516 /dist and truncation requests in 3.2 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.29 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1832 shared positions of 2 truncated texts
- ✓healthy after requests
3
error3h 20m into the runtook 2m 12sfree (stopped before any GPU time)
The self-check could not finish (ConnectionError: [Errno 111] Connect call failed ('34.155.101.90', 443)).
2 gates
- ✓review prerequisites
- ✓entry points and files
4
passed3h 22m into the runtook 3m 11s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 66 s, writing 6 files (441623251 bytes) to OUTPUT_DIR. End of its log: 3, "attn_dropout": 0.0, "emb_dropout": 0.0, "tie_embeddings": false}, "t": 12.4} {"event": "train_start", "k": 0, "grad_accum": 4, "micro_bs": 32, "seq_len": 2048, "tokens_per_step": 262144, "slice_seconds": 16.6, "t": 17.2} {"event": "step", "k": 0, "step": 1, "loss": 9.7041, "lr_mult": 0.001, "epoch": 0.016, "tok_per_s": 0, "progress": 0.0, "mem_gb": 43.67, "t": 58.9} {"event": "checkpoint", "k": 0, "step": 1, "which": "ema", "loss": 9.7036, "bpb": 4.2713, "save_seconds": 0.5, "t": 60.1} {"event": "final_eval", "k": 0, "step": 1, "loss_raw": 9.7036, "loss_ema": 9.7036, "bpb_raw": 4.2713, "bpb_ema": 4.2713, "best": {"loss": 9.703594207763672, "which": "ema", "step": 1, "epoch": 0.016}, "train_seconds": 42.9, "epochs": 0.016, "t": 60.1} {"event": "ensemble", "n_models": 1, "holdout_loss": 9.7036, "holdout_bpb": 4.2713, "singles": [9.7036], "t": 60.6} {"event": "done", "wall": 60.6, "ensemble_holdout_bpb": 4.2713, "n_models": 1, "t": 60.6} train.py exit status 0 at 2026-10-04T01:31:01Z
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 16384, "special": 2}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.963188562825735e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.3 s and 86 of 516 /dist and truncation requests in 4.2 s, each scaled to its full count, plus the first requests (1.7 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.31 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1470 shared positions of 2 truncated texts
- ✓healthy after requests

Data-efficient pretraining

Score

Verifier details