Data-efficient pretraining
- Score
- 5.5
- Wallet spend
- $31.41 of $200
- Model cost
- $9.70
- Agent time
- 4h 22m
- Steps
- 165
- Tokens in / out
- 24.9M / 137k
- Self-checks
- 1 passed of 3 used
- Rank at $200 wallet
- 11th of 14
Score
5.5raw metric 0.05505
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4494
- reward
- 0.0551
- nll_nats
- 5187591.6259
- bytes
- 5163651
- documents
- 551
- tokens
- 1487592
- unscored_documents
- 0
- vocab_size
- 6145
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. lm.train_tokenizer
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. lm.train_tokenizer builds an empty tokenizers BPE model and trains it with BpeTrainer on the corpus files. lm.CausalLM sets its weights with nn.init.normal_(std=0.02) and RMSNorm weights to ones. train.py trains this model with AdamW. No from_pretrained or other checkpoint loading appears in the code or the trajectory.
✓no embedded external dataThe shipped files are code, a manifest and logs; no weights are shipped. The only constant table is the standard GPT-2 b
The shipped files are code, a manifest and logs; no weights are shipped. The only constant table is the standard GPT-2 bytes_to_unicode byte mapping, which is an algorithm constant. The vocabulary, merges and weights are all learned from CORPUS_DIR at train time. I found no embedded word lists, statistics or encoded data.
✓training uses only corpustrain.sh runs train.py. train.py reads only *.txt files from CORPUS_DIR and writes tokenizer.json, model.safetensors, co
train.sh runs train.py. train.py reads only *.txt files from CORPUS_DIR and writes tokenizer.json, model.safetensors, config.json, train_log.jsonl and train_summary.json to OUTPUT_DIR. There is no network access and no pretrained loading. The data source is declared in architecture.json and DATA_MANIFEST.md as the task-provided corpus. During development the participant used validation.jsonl only for evaluation on the spin machine, not in the shipped training code.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py loads config.json, tokenizer.json and model.safetensors only from WEIGHTS_DIR, which ar
serve.sh runs serve.py. serve.py loads config.json, tokenizer.json and model.safetensors only from WEIGHTS_DIR, which are the files train.py writes, plus the submitted lm.py. It reads no corpus, cache or network resource.
✓causal normalized scoring/score and /dist both go through the same _logprobs_for_window function: one model forward with a banded causal mask, fo
/score and /dist both go through the same _logprobs_for_window function: one model forward with a banded causal mask, followed by log_softmax over the full vocabulary. Padding is placed after the real tokens, and the causal mask keeps it out of earlier positions. The window start for each target depends only on its position, so every log-prob depends only on the prefix, and dist uses exactly the window and row that score uses. There is no request caching and no special-casing of the audit endpoints.
✓no slowrun lookupSlowrun appears in the trajectory only in the task prompt and contract (trajectory#1, trajectory#2) and in the participa
Slowrun appears in the trajectory only in the task prompt and contract (trajectory#1, trajectory#2) and in the participant's own manifest text. The participant made no web searches or fetches about it. The only GitHub download was the s5cmd release tarball.
✓harness credentials used only by the harnessThe trajectory has no reference to API_KEY or BASE_URL variables and no launch of claude or other harness CLIs. The part
The trajectory has no reference to API_KEY or BASE_URL variables and no launch of claude or other harness CLIs. The participant's own commands made no model API calls. External activity was limited to spin compute and storage commands, a check that pypi was reachable, and the s5cmd download.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 57m into the runtook 5m 28s
startup 20.8 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 98 s, writing 5 files (632815963 bytes) to OUTPUT_DIR. End of its log: 1:03:20] total tokens=17737831 bytes=54399840 bytes/token=3.067 encode 13.3s [21:03:20] train tokens=17558838 dev tokens=178993 dev bytes=548291 [21:03:22] model params=158.1M cfg={'vocab_size': 6145, 'd_model': 1024, 'n_layer': 12, 'n_head': 16, 'd_ff': 2752, 'max_seq_len': 1024, 'dropout': 0.1, 'rope_theta': 10000.0} [21:03:25] torch.compile wrapper created (2.2s); first steps compile [21:04:08] step 25/None loss 6.6904 lr 8.33e-04 76k tok/s elapsed 66s [21:04:14] throughput 2.52 steps/s (330k tok/s); planned total_steps=90 [21:04:14] EMA auto decay=0.99000 (horizon 100 steps) [21:04:18] step 50/90 loss 5.5547 lr 1.67e-03 329k tok/s elapsed 76s [21:04:28] step 75/90 loss 5.1147 lr 1.15e-03 328k tok/s elapsed 86s [21:04:35] eval step 90: raw dev_bpb=2.2724 (nll/tok 4.8470) [21:04:35] saved best (raw) dev_bpb=2.2724 step=90 [21:04:36] eval step 90: ema dev_bpb=2.5517 [21:04:36] done: steps=90 tokens_seen=11796480 best={'bpb': 2.272406026624541, 'kind': 'raw', 'step': 90} elapsed=93s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 20.8 s
- ✓GET /vocab · {"tokens": 6145, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1499}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 989}
- ✓POST /dist · {"prefix_tokens": 749, "logprobs": 6145, "logsumexp": -1.0512005133023195e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.8 s and 86 of 516 /dist and truncation requests in 12.6 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (24 s) as measured
- ✓slowest request · 183 answered requests, median 0.56 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 901 shared positions of 2 truncated texts
- ✓healthy after requests
- 2failed3h 5m into the runtook 5m 37s
startup 35.6 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 178 s, writing 5 files (632816639 bytes) to OUTPUT_DIR. End of its log: irst steps compile [21:10:30] step 25/None loss 6.6910 lr 8.33e-04 47k tok/s elapsed 96s [21:10:36] throughput 2.51 steps/s (329k tok/s); planned total_steps=209 [21:10:36] EMA auto decay=0.99000 (horizon 100 steps) [21:10:40] step 50/209 loss 5.5786 lr 1.67e-03 328k tok/s elapsed 106s [21:10:50] step 75/209 loss 5.1642 lr 1.96e-03 328k tok/s elapsed 116s [21:11:00] step 100/209 loss 4.8205 lr 1.70e-03 328k tok/s elapsed 126s [21:11:10] step 125/209 loss 4.6600 lr 1.26e-03 327k tok/s elapsed 136s [21:11:20] step 150/209 loss 4.4604 lr 7.64e-04 327k tok/s elapsed 146s [21:11:30] step 175/209 loss 4.3650 lr 3.47e-04 327k tok/s elapsed 156s [21:11:40] step 200/209 loss 4.2895 lr 1.21e-04 327k tok/s elapsed 166s [21:11:45] eval step 209: raw dev_bpb=1.9948 (nll/tok 4.2548) [21:11:45] saved best (raw) dev_bpb=1.9948 step=209 [21:11:46] eval step 209: ema dev_bpb=2.0909 [21:11:46] done: steps=209 tokens_seen=27394048 best={'bpb': 1.994757540109802, 'kind': 'raw', 'step': 209} elapsed=172s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 35.6 s
- ✓GET /vocab · {"tokens": 6145, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1499}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 989}
- ✓POST /dist · {"prefix_tokens": 749, "logprobs": 6145, "logsumexp": 1.1729550397632238e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 13.0 s and 86 of 516 /dist and truncation requests in 23.8 s, each scaled to its full count, plus the first requests (0.5 s) and the restart and its first /dist (47 s) as measured
- ✓slowest request · 183 answered requests, median 1.28 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4437 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed3h 15m into the runtook 6m
startup 29.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 177 s, writing 5 files (632816775 bytes) to OUTPUT_DIR. End of its log: ok/s elapsed 79s [21:20:48] throughput 2.49 steps/s (326k tok/s); planned total_steps=246 [21:20:48] EMA auto decay=0.99000 (horizon 100 steps) [21:20:53] step 50/246 loss 5.5425 lr 1.67e-03 324k tok/s elapsed 89s [21:21:03] step 75/246 loss 5.0896 lr 1.97e-03 322k tok/s elapsed 100s [21:21:13] step 100/246 loss 4.7748 lr 1.80e-03 322k tok/s elapsed 110s [21:21:23] step 125/246 loss 4.6323 lr 1.50e-03 321k tok/s elapsed 120s [21:21:33] step 150/246 loss 4.4393 lr 1.11e-03 320k tok/s elapsed 130s [21:21:44] step 175/246 loss 4.3413 lr 7.20e-04 320k tok/s elapsed 140s [21:21:54] step 200/246 loss 4.2392 lr 3.84e-04 319k tok/s elapsed 151s [21:22:04] step 225/246 loss 4.1558 lr 1.65e-04 319k tok/s elapsed 161s [21:22:13] eval step 246: raw dev_bpb=1.9338 (nll/tok 4.1248) [21:22:14] saved best (raw) dev_bpb=1.9338 step=246 [21:22:14] eval step 246: ema dev_bpb=2.0089 [21:22:14] done: steps=246 tokens_seen=32243712 best={'bpb': 1.9338081092885746, 'kind': 'raw', 'step': 246} elapsed=171s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 29.2 s
- ✓GET /vocab · {"tokens": 6145, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1499}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 989}
- ✓POST /dist · {"prefix_tokens": 749, "logprobs": 6145, "logsumexp": -8.289941710870607e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 12.5 s and 86 of 516 /dist and truncation requests in 18.5 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (35 s) as measured
- ✓slowest request · 183 answered requests, median 1.35 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2115 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output/ (11 files, code only, no weights):
Solution
- Tokenizer: byte-level BPE trained from scratch on the corpus inside train.py, 6144 tokens (+BOS), 3.07 bytes/token, verified exact byte round-trip on all inputs.
- Model (lm.py): 158M-param decoder-only Transformer — d1024, 12 layers, RMSNorm + QK-norm, RoPE, SwiGLU, tied embeddings, dropout 0.1, context 1024.
- Training: AdamW + cosine LR (2e-3), 131k-token batches of random crops with BOS starts, bf16 autocast + torch.compile (~326k tok/s on H100).
train.pybenchmarks throughput on the machine it runs on and plans total steps to fillTRAIN_BUDGET_SECONDS, so it adapts to grading hardware. Best checkpoint (raw vs EMA) is selected on an internal holdout (last 1% of tokens of each corpus file, excluded from training). No public validation/test data was used for training or selection; all tuning used that internal split. - Serving (serve.py): stdlib HTTP server, fp32 inference.
/scoreand/distshare an exact chunk-aligned window definition with banded causal attention, so they agree to 0.0 nats (an earlier window-shift numeric mismatch of 0.06 nats was found by selfcheck, root-caused, and fixed).
Verified results
- Full 60-min dress rehearsal of the exact submitted
train.shon an H100: finished in 3512s, internal-holdout BPB 1.2727 (a later 208M-param L16 variant scored worse, 1.2838, and was rejected). - Server: startup ~29s (limit 300s), 169 public docs scored in 24s, concurrency and restart tested.
frontier selfcheck(3rd run, on the exact submitted digest): PASSED all gates — train probe exit 0, score/dist 0 nats, truncation 0 nats, projected evaluation 223s of 3600s, slowest request 2.74s of 120s.- Submission confirmed at 2026-10-12 22:25 UTC.