Data-efficient pretraining
- Score
- 5.8
- Wallet spend
- $7.68 of $10
- Model cost
- $0.62
- Agent time
- 1h 42m
- Steps
- 254
- Tokens in / out
- 26.1M / 278k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 13th of 19
Score
5.8raw metric 0.05821
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4487
- reward
- 0.0582
- nll_nats
- 5185203.9721
- bytes
- 5163651
- documents
- 551
- tokens
- 1325961
- unscored_documents
- 0
- vocab_size
- 16001
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In src/data.py, bui
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. In src/data.py, build_tokenizer creates a fresh byte-level BPE with tokenizers.BpeTrainer and trains it on the corpus texts. In src/model.py, TransformerLM sets every weight with normal(0, 0.02) and scales the residual output layers down. src/train.py builds this model and trains it with AdamW. No code anywhere calls from_pretrained or loads a pretrained checkpoint. The only path to an existing tokenizer is the optional TOKENIZER_CACHE dev option, and train.sh does not use it.
✓no embedded external dataThe submission is code and logs only, 29 files totalling 128.7 KB (trajectory#252). It ships no weight files or vocabula
The submission is code and logs only, 29 files totalling 128.7 KB (trajectory#252). It ships no weight files or vocabularies. bytelevel.py works out the byte-to-character map from the tokenizer at run time instead of hard-coding a table. Its only constants are UTF-8 byte ranges, which are algorithm constants. The tokenizer, weights and vocab_bytes.json are all written by train.py at run time. The only embedded text is short test and probe strings.
✓training uses only corpustrain.sh runs src/train.py with CORPUS_DIR and OUTPUT_DIR. read_corpus reads only the *.txt files in corpus_dir. Every o
train.sh runs src/train.py with CORPUS_DIR and OUTPUT_DIR. read_corpus reads only the *.txt files in corpus_dir. Every output (tokenizer.json, model.safetensors, config.json, vocab_bytes.json, train log) is written under out_dir. The data and tokenizer cache paths are optional, empty by default, and not set by train.sh. The code makes no network calls and loads no pretrained library assets. In the trajectory, the dev-box corpus was copied from /task/public/corpus through Spin storage (trajectory#54-57), and that was the task corpus.
✓serving uses only replay outputsserve.sh runs src/serve.py. Server.__init__ reads only config.json, tokenizer.json and model.safetensors from WEIGHTS_DI
serve.sh runs src/serve.py. Server.__init__ reads only config.json, tokenizer.json and model.safetensors from WEIGHTS_DIR, and it loads the weights into the submitted TransformerLM with a strict key check. The server reads no corpus, no cache and no network resource.
✓causal normalized scoring/score and /dist both go through src/scoring.py, which feeds [BOS]+ids into the same model with a band-causal mask (band
/score and /dist both go through src/scoring.py, which feeds [BOS]+ids into the same model with a band-causal mask (banded_attention in src/model.py). Row r can only see rows [r+1-C, r], so each log-prob depends only on the prefix. /dist returns log_softmax over the model's full vocab_size output, so the distribution is normalized. Requests are independent: there is no request-history cache and no lookup of previously seen text, and the audit endpoints get no special handling. The server only takes a lock while running the model.
✓no slowrun lookupThe only Slowrun mentions in the trajectory are the task prompt and the contract text, plus summaries of it (trajectory#
The only Slowrun mentions in the trajectory are the task prompt and the contract text, plus summaries of it (trajectory#1). There are no WebSearch or WebFetch calls and no git clone. The curl commands in the session hit github.com headers, pypi, the s5cmd release, and localhost server tests. None of them targets Slowrun.
✓harness credentials used only by the harnessNo step in the trajectory reads, echoes or uses ANTHROPIC, OPENAI, OPENROUTER or harness *_BASE_URL credentials. No othe
No step in the trajectory reads, echoes or uses ANTHROPIC, OPENAI, OPENROUTER or harness *_BASE_URL credentials. No other harness CLI sessions were launched. The object-storage credentials used in trajectory#53-57 are Spin's own wallet-scoped transfer credentials, which the participant is allowed to use.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 11m into the runtook 8sfree (stopped before any GPU time)
failed review prerequisites: architecture.json is not valid JSON: Expecting ',' delimiter: line 28 column 18 (char 1486)
1 gates
- ✗review prerequisites · architecture.json is not valid JSON: Expecting ',' delimiter: line 28 column 18 (char 1486)
- 2passed1h 13m into the runtook 4m 34s
startup 4.1 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 190 s, writing 5 files (68409986 bytes) to OUTPUT_DIR. End of its log: ent=step step=500 total_steps=652 loss=3.2264 lr=0.000339 tok_per_s=656790 elapsed=99.9 eta=30.4 event=step step=525 total_steps=652 loss=3.1847 lr=0.000264 tok_per_s=663103 elapsed=103.9 eta=25.1 event=step step=550 total_steps=652 loss=3.2346 lr=0.000199 tok_per_s=668926 elapsed=107.9 eta=20.0 event=step step=575 total_steps=652 loss=3.2133 lr=0.000147 tok_per_s=674341 elapsed=111.9 eta=15.0 event=step step=600 total_steps=652 loss=3.1755 lr=0.000109 tok_per_s=679390 elapsed=115.9 eta=10.0 event=step step=625 total_steps=652 loss=3.2286 lr=8.4e-05 tok_per_s=684094 elapsed=119.9 eta=5.2 event=step step=650 total_steps=652 loss=3.2931 lr=7.5e-05 tok_per_s=688501 elapsed=123.9 eta=0.4 event=step step=652 total_steps=652 loss=3.2072 lr=7.5e-05 tok_per_s=688836 elapsed=124.2 eta=0.0 event=train_done steps=652 epochs=5.227 train_seconds=124.2 stop_reason=planned tokens_seen=85542400 final_lr=7.5e-05 event=saved seconds=0.08 output_dir=/replay/output total_seconds=181.5 stop_reason=planned
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.1 s
- ✓GET /vocab · {"tokens": 16001, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 960}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16001, "logsumexp": -4.044126145075211e-09, "score_minus_dist": 1.1920928955078125e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.2 s and 86 of 516 /dist and truncation requests in 3.1 s, each scaled to its full count, plus the first requests (1.5 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.17 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1108 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed1h 36m into the runtook 4m 25s
startup 3.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 190 s, writing 5 files (68409572 bytes) to OUTPUT_DIR. End of its log: 8 event=step step=425 total_steps=600 loss=3.2012 lr=0.000485 tok_per_s=634612 elapsed=87.9 eta=36.2 event=step step=450 total_steps=600 loss=3.3139 lr=0.000386 tok_per_s=641998 elapsed=92.0 eta=30.7 event=step step=475 total_steps=600 loss=3.2242 lr=0.000297 tok_per_s=648670 elapsed=96.1 eta=25.3 event=step step=500 total_steps=600 loss=3.244 lr=0.00022 tok_per_s=654248 elapsed=100.3 eta=20.1 event=step step=525 total_steps=600 loss=3.2037 lr=0.000159 tok_per_s=659870 elapsed=104.4 eta=14.9 event=step step=550 total_steps=600 loss=3.2575 lr=0.000113 tok_per_s=664821 elapsed=108.5 eta=9.9 event=step step=575 total_steps=600 loss=3.2435 lr=8.5e-05 tok_per_s=669690 elapsed=112.6 eta=4.9 event=step step=600 total_steps=600 loss=3.2043 lr=7.5e-05 tok_per_s=674031 elapsed=116.8 eta=0.0 event=train_done steps=600 epochs=4.81 train_seconds=116.8 stop_reason=planned tokens_seen=78720000 final_lr=7.5e-05 event=saved seconds=0.07 output_dir=/replay/output total_seconds=182.9 stop_reason=planned
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.6 s
- ✓GET /vocab · {"tokens": 16001, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 960}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16001, "logsumexp": -1.2437090268768713e-07, "score_minus_dist": 4.291534423828125e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.8 s and 86 of 516 /dist and truncation requests in 2.2 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.14 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2281 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-20 21:15:33 UTC.
Final submission (/workspace/output, 29 files, 129 KB):
| tokenizer | byte-level BPE, 16 000 merges, trained on the corpus at run time (3.33 bytes/token) |
| model | 6 layers, 6 heads, d_model 384, d_ff 1024, ctx 1024, RMSNorm/RoPE/SwiGLU, tied embeddings, 16.8 M params |
| training | 30 epochs (3 742 steps, 490 M tokens), AdamW, cosine from 1.5e-3, dropout 0.2 — 664 s on an H100 |
| dev bits/byte | 1.0188 on a held-out 2 % tail slice of the corpus itself |
Key points from this final stretch:
- Selection finished on corpus-only data. The three 30-epoch curves gave dropout 0.1 → 1.0372, 0.2 → 1.0188, 0.3 → 1.0211, so dropout 0.2 won;
train.shnow defaults to it withMAX_EPOCHS=30. - Submission-exact run on the box:
bash train.shwith the contract's env vars,CORPUS_DIR/OUTPUT_DIR/TRAIN_BUDGET_SECONDS=3600/REPLAY_START, dev-frac 0 → exit 0 in 664 s, weights + tokenizer + config written (logs/train_log_final.jsonl). - Serving verified against that model:
/health0.4 s, worst|dist − score|1.0e-5 nats, truncation 1.1e-5, future-leak 5e-6, 8 requests in flight in 1.0 s with zero drift → audit PASS. It also serves correctly from a read-onlyWEIGHTS_DIRwith an empty scratch. frontier selfcheckpassed twice (the first attempt caught an invalidarchitecture.json, which I fixed and re-validated): 8/8 gates, startup 3.6 s of 300 s, slowest request 0.61 s of 120 s, evaluation projected at 33.5 s of 3600 s.
One caveat worth stating plainly: the public validation set shipped with the task was never used for training, calibration or selection, and babylm/validate.py could not be imported in either the sandbox or the rented box, so the task's own validate.py number was never produced locally — frontier selfcheck exercised the equivalent response checks instead.