Data-efficient pretraining
- Score
- 60.1
- Wallet spend
- $8.91 of $10
- Model cost
- $1.00
- Agent time
- 4h 46m
- Steps
- 317
- Tokens in / out
- 31.3M / 325k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 3rd of 19
Score
60.1raw metric 0.601
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3342
- reward
- 0.6010
- nll_nats
- 4775305.5228
- bytes
- 5163651
- documents
- 551
- tokens
- 1321054
- unscored_documents
- 0
- vocab_size
- 16385
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratchsrc/model.py defines a decoder-only transformer (RMSNorm, RoPE, SwiGLU, tied embeddings). Its weights are initialized ra
src/model.py defines a decoder-only transformer (RMSNorm, RoPE, SwiGLU, tied embeddings). Its weights are initialized randomly from a normal distribution with std 0.02. src/train.py builds each ensemble member with make_model after torch.manual_seed, and nothing loads a pretrained checkpoint. There are no from_pretrained calls and no checkpoint reads before training. The tokenizer is a byte-level BPE trained from scratch on the corpus with tokenizers' BpeTrainer in src/common.py. architecture.json declares base_model none / from-scratch. The trajectory shows no download of pretrained weights.
✓no embedded external dataThe shipped source holds only code, hyperparameters and the algorithmic GPT-2 bytes_to_unicode mapping in src/common.py.
The shipped source holds only code, hyperparameters and the algorithmic GPT-2 bytes_to_unicode mapping in src/common.py. It contains no embedded corpora, vocabularies, merges or weight blobs. The submission is 17 files totaling 94.8 KB, and the logs ship only as provenance.
✓training uses only corpustrain.sh runs src/prepare.py --corpus-dir $CORPUS_DIR and then src/train.py on prep outputs in OUTPUT_DIR. read_corpus_f
train.sh runs src/prepare.py --corpus-dir $CORPUS_DIR and then src/train.py on prep outputs in OUTPUT_DIR. read_corpus_files in src/common.py reads only *.txt files from the corpus directory. The held-out selection block is a contiguous 3% split carved out of the corpus training files. DATA_MANIFEST.md and architecture.json declare the source as the provided BabyLM strict-small train files, with SHA-256 hashes, and state that no external data or downloads were used. validation.jsonl was used only in the agent's development evaluations and is never read by the shipped train.sh.
✓serving uses only replay outputsserve.sh sets WEIGHTS_DIR (default /replay/output) and execs src/serve.py. Engine loads config.json, tokenizer.json and
serve.sh sets WEIGHTS_DIR (default /replay/output) and execs src/serve.py. Engine loads config.json, tokenizer.json and model_*.safetensors from WEIGHTS_DIR only. Serving makes no network calls and uses no external models.
✓causal normalized scoringsrc/model.py uses SDPA with is_causal=True. src/serve.py computes log_softmax over the full 16385-token vocabulary for e
src/model.py uses SDPA with is_causal=True. src/serve.py computes log_softmax over the full 16385-token vocabulary for each ensemble member and combines them with logsumexp - log(n), which yields a normalized mixture. /score and /dist share the same fixed _window/_win_len windowing, and no request state is cached across calls. The final selfcheck reported 0-nat differences for normalization and for /score vs /dist agreement (trajectory#317).
✓no slowrun lookupSlowrun is mentioned only in the task prompt/contract (trajectory#1, trajectory#2) and in the agent restating constraint
Slowrun is mentioned only in the task prompt/contract (trajectory#1, trajectory#2) and in the agent restating constraints. No WebSearch, WebFetch, git clone or GitHub access to qlabs-eng/slowrun appears in the trajectory.
✓harness credentials used only by the harnessNo ANTHROPIC key, harness base URL or printenv of credentials appears in participant commands. Network use was limited t
No ANTHROPIC key, harness base URL or printenv of credentials appears in participant commands. Network use was limited to pip package installs and to the participant's own Spin S3 session for moving code and corpus tarballs (trajectory#37, trajectory#235), which is permitted. No additional harness sessions or model API calls were made.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed42m 28s into the runtook 7m 24s
startup 2.4 s of 300 s; 6/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 180 s, writing 9 files (221672190 bytes) to OUTPUT_DIR. End of its log: s, 2.9s [prepare] gutenberg.train.txt: 3518276 tokens, 21 val blocks, 2.8s [prepare] open_subtitles.train.txt: 3692878 tokens, 22 val blocks, 2.6s [prepare] simple_wiki.train.txt: 2312398 tokens, 14 val blocks, 1.8s [prepare] switchboard.train.txt: 41345 tokens, 1 val blocks, 0.4s [prepare] done: train_tokens=16175436 val_blocks=98 bytes/token=3.324 total=15.3s [train.sh] prep took 15s, 185s left for training [train] train tokens 16.18M, val blocks 98 [train m0] params=46.9M non-emb=38.5M L=2048 chunk=512 batch=16 [train] torch.compile enabled [train m0] first step took 25.0s (compile+init) [train m0] step 1 t=25s frac=0.00 lr=7.51e-05 loss=9.7651 val_bpb=3.9772 tok/s=1313 [train m0] step 2593 t=145s frac=0.94 lr=1.65e-04 loss=2.9990 val_bpb=1.3542 tok/s=586026 [train m0] stopping after 2801 steps, 91.8M tokens (5.7 epochs) in 155s [train m0] final val_bpb raw=1.3461 ema=1.4118 [train m0] saved raw -> /replay/output/model_0.safetensors [train] wrote config.json [train.sh] done in 179s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.4 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1307}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 957}
- ✓POST /dist · {"prefix_tokens": 653, "logprobs": 16385, "logsumexp": 1.8121528500003592e-08, "score_minus_dist": 0.0010192394256591797}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.0 s and 86 of 516 /dist and truncation requests in 2.4 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.20 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✗truncated text · max difference over 2211 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed1h 14m into the runtook 2m 37s
startup 2.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 102 s, writing 9 files (221672297 bytes) to OUTPUT_DIR. End of its log: 276 tokens, 21 val blocks, 2.1s [prepare] open_subtitles.train.txt: 3692878 tokens, 22 val blocks, 2.2s [prepare] simple_wiki.train.txt: 2312398 tokens, 14 val blocks, 1.3s [prepare] switchboard.train.txt: 41345 tokens, 1 val blocks, 0.2s [prepare] done: train_tokens=16175436 val_blocks=98 bytes/token=3.324 total=11.6s [train.sh] prep took 12s, 188s left for training [train] train tokens 16.18M, val blocks 98 [train] wall clock selects 1 model(s) at ~470s each [train m0] params=46.9M non-emb=38.5M L=2048 chunk=1024 batch=8 [train] torch.compile enabled [train m0] first step took 23.3s (compile+init) [train m0] step 1 t=23s frac=0.00 lr=7.52e-05 loss=9.7551 val_bpb=3.9833 ema=4.3018 best=3.9833 tok/s=702 [train m0] stopping after 2149 steps, 35.2M tokens (2.2 epochs) in 78s [train m0] final val_bpb raw=1.5364 ema=1.6277 best_raw=3.9833 best_ema=4.3018 -> final_raw (1.5364) [train m0] saved final_raw -> /replay/output/model_0.safetensors [train] wrote config.json [train.sh] done in 102s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.5 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1307}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 957}
- ✓POST /dist · {"prefix_tokens": 653, "logprobs": 16385, "logsumexp": -3.5404208453826413e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.0 s and 86 of 516 /dist and truncation requests in 1.4 s, each scaled to its full count, plus the first requests (0.4 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 183 answered requests, median 0.12 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4365 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed1h 28m into the runtook 4m 21s
startup 3.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 175 s, writing 9 files (221672280 bytes) to OUTPUT_DIR. End of its log: btitles.train.txt: 3692878 tokens, 22 val blocks, 3.5s [prepare] simple_wiki.train.txt: 2312398 tokens, 14 val blocks, 2.5s [prepare] switchboard.train.txt: 41345 tokens, 1 val blocks, 0.4s [prepare] done: train_tokens=16175436 val_blocks=98 bytes/token=3.324 total=21.1s [train.sh] prep took 28s, 172s left for training [train] train tokens 16.18M, val blocks 98 [train] wall clock selects 5 model(s) at ~30s each [train m0] params=46.9M non-emb=38.5M L=2048 chunk=1024 batch=8 [train] torch.compile enabled [train m0] first step took 37.6s (compile+init) [train m0] step 1 t=38s frac=0.00 lr=7.73e-05 loss=9.7554 val_bpb=3.9747 ema=4.3018 best=3.9747 tok/s=435 [train m0] stopping after 1 steps, 0.0M tokens (0.0 epochs) in 41s [train m0] final val_bpb raw=3.9747 ema=4.3018 best_raw=3.9747 best_ema=4.3018 -> final_raw (3.9747) [train m0] saved final_raw -> /replay/output/model_0.safetensors [train] stopping early: -36s left, 4 model(s) skipped [train] wrote config.json [train.sh] done in 175s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.0 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1307}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 957}
- ✓POST /dist · {"prefix_tokens": 653, "logprobs": 16385, "logsumexp": -1.2421084338853916e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.9 s and 86 of 516 /dist and truncation requests in 1.8 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.17 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1056 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed4h 44m into the runtook 3m 3s
startup 6.1 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 102 s, writing 11 files (597195660 bytes) to OUTPUT_DIR. End of its log: irst step took 0.2s (compile+init) [train m1] step 1 t=0s frac=0.00 lr=7.62e-05 loss=9.8056 val_bpb=3.9911 ema=4.2692 best=3.9911 tok/s=85904 [train m1] stopping after 285 steps, 4.7M tokens (0.3 epochs) in 10s [train m1] final val_bpb raw=2.0453 ema=3.3851 best_raw=3.9911 best_ema=4.2692 -> final_raw (2.0453) [train m1] saved final_raw -> /replay/output/model_1.safetensors [train m2] params=46.9M non-emb=38.5M L=2048 chunk=1024 batch=8 [train] torch.compile enabled [train m2] first step took 0.1s (compile+init) [train m2] step 1 t=0s frac=0.00 lr=7.66e-05 loss=9.7582 val_bpb=3.9558 ema=4.2414 best=3.9558 tok/s=248266 [train m2] stopping after 287 steps, 4.7M tokens (0.3 epochs) in 10s [train m2] final val_bpb raw=2.0157 ema=3.4054 best_raw=3.9558 best_ema=4.2414 -> final_raw (2.0157) [train m2] saved final_raw -> /replay/output/model_2.safetensors [train] stopping early: 19s left, 3 model(s) skipped [train] wrote config.json [train] ensemble(3) val_bpb=2.12918 [train.sh] done in 102s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.1 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1307}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 957}
- ✓POST /dist · {"prefix_tokens": 653, "logprobs": 16385, "logsumexp": -1.7814663122450725e-07, "score_minus_dist": -1.2088337353333145e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.3 s and 86 of 516 /dist and truncation requests in 3.0 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.41 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1669 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 00:19:54 UTC. Final state of the hand-in:
What was submitted — 17 files (94.8 KB) in /workspace/output/:
train.sh→ trains a byte-level BPE tokenizer (16,384 merges + BOS, 3.32 bytes/token) and a time-budgeted ensemble of up to 6 LLaMA-style 47M-param transformers (RMSNorm/RoPE/SwiGLU, tied embeddings, d_model 512 × 12 layers), bf16 autocast + fp32 master weights,torch.compile, AdamW with dropout 0.35 / weight decay 0.4 / grad clip 1.0 / EMA 0.9995. It fits prep + all members into the wall clock via a self-timed budget and picks the best of {final raw, final EMA, best raw, best EMA} per member on held-out documents.serve.sh→ loads those checkpoints and answers/health,/vocab,/score,/distfrom an ensemble-averaged (logsumexp) distribution, using fixed-shape causally masked windows so/scoreand/distagree exactly.src/,logs/,DATA_MANIFEST.md,architecture.json— corpus provenance, the honest held-out selection methodology, and logs from a real run of the shipped code.
Final selfcheck (tree digest 4224c7a3…, H100 80GB HBM3): passed 8/8 gates — train.sh exited 0, startup 6.1 s of 300 s, restart 4.2 s, slowest request 0.83 s, projected evaluation time 65.7 s of 3600 s, and /dist normalization, /score vs /dist agreement, bias, and truncation differences all 0 nats.
The key late fix was swapping the internal validation from scattered held-out chunks to a contiguous 3 % region per corpus file, scored exactly as the server scores a document; the old proxy was inflated by memorization (1.14 internal vs 1.34 on the task's validation), and the regularization strength plus EMA decay were then re-tuned on the clean proxy (best single model 1.245 bpb).