Data-efficient pretraining
- Score
- 38.9
- Wallet spend
- $55.84 of $200
- Model cost
- $0.87
- Agent time
- 3h 54m
- Steps
- 289
- Tokens in / out
- 29.1M / 306k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 10th of 19
Score
38.9raw metric 0.3887
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3790
- reward
- 0.3887
- nll_nats
- 4935622.0624
- bytes
- 5163651
- documents
- 551
- tokens
- 1256041
- unscored_documents
- 0
- vocab_size
- 32769
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py builds the
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py builds the byte-level BPE tokenizer with tokenizers' BpeTrainer.train_from_iterator, using only the corpus documents. It creates each GPT model with GPT(mcfg), which model.py initializes randomly (normal_(0, 0.02) plus scaled output projections), and then trains it with AdamW. Nothing calls from_pretrained or loads any checkpoint during training. The trajectory shows the final runs (trajectory#279-283) training model_0 to model_3 from scratch through train.py.
✓no embedded external dataThe shipped files are Python and shell source code, JSON logs and a manifest. They contain no vocabularies, merge lists,
The shipped files are Python and shell source code, JSON logs and a manifest. They contain no vocabularies, merge lists, weights or tables. The vocabulary is learned at run time. The only constants are hyperparameters, the standard GPT-2 byte-to-unicode mapping (an algorithm constant), RoPE base values and one short tokenizer round-trip test string.
✓training uses only corpustrain.sh runs train.py, which reads only glob(CORPUS_DIR/*.txt) and writes the tokenizer, vocab, config, safetensors, mo
train.sh runs train.py, which reads only glob(CORPUS_DIR/*.txt) and writes the tokenizer, vocab, config, safetensors, models.json and train_log.jsonl only to OUTPUT_DIR. It makes no network calls and loads no pretrained model or tokenizer. During development the participant tokenized validation.jsonl to count its tokens and scored it, but the shipped training path never reads it. DATA_MANIFEST.md declares the task corpus as the data source.
✓serving uses only replay outputsserve.sh runs server.py, which loads only config.json, vocab.json, tokenizer.json, models.json and the listed safetensor
serve.sh runs server.py, which loads only config.json, vocab.json, tokenizer.json, models.json and the listed safetensors files, all from WEIGHTS_DIR, together with the submitted model.py and scoring.py. It reads no corpus, cache or network resource.
✓causal normalized scoring/score and /dist both use the same ensemble of 4 trained causal GPTs. Attention uses is_causal=True. The windows are anc
/score and /dist both use the same ensemble of 4 trained causal GPTs. Attention uses is_causal=True. The windows are anchored on a fixed grid that depends only on token position, with right padding that causal masking makes irrelevant. /dist applies a full-vocabulary log-softmax (including the BOS id) and mixes the models as the log of their mean probability, so the result is normalized. There is no request caching, no lookup of previously seen text and no special-casing of audit requests. The self-check reported zero normalization and score-versus-dist error (trajectory#277).
✓no slowrun lookupThe trajectory contains no WebSearch or WebFetch calls; its tool calls are only Bash, Write, Edit and Read. Every mentio
The trajectory contains no WebSearch or WebFetch calls; its tool calls are only Bash, Write, Edit and Read. Every mention of Slowrun is the task instructions or a contract quote. The only GitHub access was a download of the s5cmd release, and there is no clone or fetch of qlabs-eng/slowrun or related pages.
✓harness credentials used only by the harnessSearches of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER API keys or base URLs, and no launch of extra cla
Searches of the trajectory found no use of ANTHROPIC/OPENAI/OPENROUTER API keys or base URLs, and no launch of extra claude, codex or similar CLI sessions. The credentials the participant handled were Spin's own S3 transfer session credentials (trajectory#66), which are not harness credentials. Compute ran through the spin CLI.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed30m 21s into the runtook 16sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed31m 26s into the runtook 8m 13s
startup 3.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 199 s, writing 5 files (443327566 bytes) to OUTPUT_DIR. End of its log: .8783, "lr": 0.00015, "phase": "step", "step": 230, "t": 167.31, "tok_per_s": 266580.9, "tokens": 30146560} {"loss": 4.1507, "lr": 0.000119, "phase": "step", "step": 240, "t": 172.23, "tok_per_s": 266574.2, "tokens": 31457280} {"loss": 3.9177, "lr": 9.3e-05, "phase": "step", "step": 250, "t": 177.16, "tok_per_s": 266543.8, "tokens": 32768000} {"loss": 3.9509, "lr": 7.3e-05, "phase": "step", "step": 260, "t": 182.09, "tok_per_s": 266523.1, "tokens": 34078720} {"loss": 3.9565, "lr": 5.9e-05, "phase": "step", "step": 270, "t": 187.01, "tok_per_s": 266509.0, "tokens": 35389440} {"loss": 4.016, "lr": 5.2e-05, "phase": "step", "step": 280, "t": 191.93, "tok_per_s": 266506.9, "tokens": 36700160} {"loss": 4.126, "lr": 5e-05, "phase": "step", "step": 287, "t": 195.01, "tok_per_s": 266498.8, "tokens": 37519360} {"phase": "train_done", "seconds": 140.8, "steps": 287, "t": 195.01, "tokens": 37519360, "tokens_per_s": 266498.3} {"phase": "saved", "seconds": 0.4, "t": 195.42, "total_seconds": 195.4}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.3 s
- ✓GET /vocab · {"tokens": 32769, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32769, "logsumexp": 4.796628949588211e-08, "score_minus_dist": -0.0010120272636413574}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.5 s and 86 of 516 /dist and truncation requests in 2.9 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 183 answered requests, median 0.21 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4127 shared positions of 2 truncated texts
- ✓healthy after requests
- 3failed2h 52m into the runtook 4m 3s
startup 2.4 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 200 s, writing 5 files (169603712 bytes) to OUTPUT_DIR. End of its log: s": 3.5016, "lr": 8e-05, "phase": "step", "step": 490, "t": 178.96, "tok_per_s": 475217.7, "tokens": 64225280} {"loss": 3.472, "lr": 7.1e-05, "phase": "step", "step": 500, "t": 181.72, "tok_per_s": 475206.9, "tokens": 65536000} {"loss": 3.425, "lr": 6.3e-05, "phase": "step", "step": 510, "t": 184.48, "tok_per_s": 475184.8, "tokens": 66846720} {"loss": 3.3573, "lr": 5.7e-05, "phase": "step", "step": 520, "t": 187.25, "tok_per_s": 475166.8, "tokens": 68157440} {"loss": 3.5226, "lr": 5.3e-05, "phase": "step", "step": 530, "t": 190.0, "tok_per_s": 475181.0, "tokens": 69468160} {"loss": 3.3825, "lr": 5.1e-05, "phase": "step", "step": 540, "t": 192.75, "tok_per_s": 475200.1, "tokens": 70778880} {"loss": 3.156, "lr": 5e-05, "phase": "step", "step": 549, "t": 195.09, "tok_per_s": 475235.2, "tokens": 71892992} {"phase": "train_done", "seconds": 151.3, "steps": 549, "t": 195.09, "tokens": 71892992, "tokens_per_s": 475234.5} {"phase": "saved", "seconds": 0.2, "t": 195.29, "total_seconds": 195.3}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.4 s
- ✓GET /vocab · {"tokens": 32769, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32769, "logsumexp": -5.125549185081013e-08, "score_minus_dist": 0.002771735191345215}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.4 s and 86 of 516 /dist and truncation requests in 2.8 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 183 answered requests, median 0.16 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5424 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed3h 0m into the runtook 4m 31s
startup 3.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 200 s, writing 5 files (169603475 bytes) to OUTPUT_DIR. End of its log: : 3.3458, "lr": 7.8e-05, "phase": "step", "step": 470, "t": 179.94, "tok_per_s": 471156.1, "tokens": 61603840} {"loss": 3.689, "lr": 6.9e-05, "phase": "step", "step": 480, "t": 182.73, "tok_per_s": 471161.0, "tokens": 62914560} {"loss": 3.5519, "lr": 6.2e-05, "phase": "step", "step": 490, "t": 185.5, "tok_per_s": 471184.7, "tokens": 64225280} {"loss": 3.5225, "lr": 5.6e-05, "phase": "step", "step": 500, "t": 188.28, "tok_per_s": 471180.7, "tokens": 65536000} {"loss": 3.4741, "lr": 5.2e-05, "phase": "step", "step": 510, "t": 191.06, "tok_per_s": 471210.3, "tokens": 66846720} {"loss": 3.4056, "lr": 5e-05, "phase": "step", "step": 520, "t": 193.83, "tok_per_s": 471230.0, "tokens": 68157440} {"loss": 3.3928, "lr": 5e-05, "phase": "step", "step": 525, "t": 195.08, "tok_per_s": 471239.7, "tokens": 68747264} {"phase": "train_done", "seconds": 145.9, "steps": 525, "t": 195.08, "tokens": 68747264, "tokens_per_s": 471238.7} {"phase": "saved", "seconds": 0.2, "t": 195.32, "total_seconds": 195.3}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.0 s
- ✓GET /vocab · {"tokens": 32769, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32769, "logsumexp": 4.951523480123399e-08, "score_minus_dist": 3.337860107421875e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.4 s and 86 of 516 /dist and truncation requests in 3.0 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.19 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2053 shared positions of 2 truncated texts
- ✓healthy after requests
- 5passed3h 21m into the runtook 4m 27s
startup 5.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 160 s, writing 9 files (669919198 bytes) to OUTPUT_DIR. End of its log: 438, "phase": "step", "step": 50, "t": 145.41, "tok_per_s": 470587.7, "tokens": 6553600} {"index": 3, "loss": 5.3235, "lr": 0.000267, "phase": "step", "step": 60, "t": 148.19, "tok_per_s": 470837.7, "tokens": 7864320} {"index": 3, "loss": 5.1981, "lr": 0.000135, "phase": "step", "step": 70, "t": 150.96, "tok_per_s": 471058.0, "tokens": 9175040} {"index": 3, "loss": 5.3462, "lr": 6.1e-05, "phase": "step", "step": 80, "t": 153.75, "tok_per_s": 471049.4, "tokens": 10485760} {"index": 3, "loss": 5.1268, "lr": 5e-05, "phase": "step", "step": 85, "t": 155.0, "tok_per_s": 471017.0, "tokens": 11075584} {"index": 3, "phase": "train_done", "seconds": 23.5, "steps": 85, "t": 155.0, "tokens": 11075584, "tokens_per_s": 471007.4} {"file": "model_3.safetensors", "index": 3, "phase": "saved", "seconds": 0.2, "t": 155.18, "total_seconds": 155.2} {"models": ["model_0.safetensors", "model_1.safetensors", "model_2.safetensors", "model_3.safetensors"], "phase": "done", "t": 155.45, "total_seconds": 155.5}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.0 s
- ✓GET /vocab · {"tokens": 32769, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32769, "logsumexp": 1.3266008291878961e-08, "score_minus_dist": -7.228061491559856e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.1 s and 86 of 516 /dist and truncation requests in 6.2 s, each scaled to its full count, plus the first requests (2.3 s) and the restart and its first /dist (12 s) as measured
- ✓slowest request · 183 answered requests, median 0.54 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4683 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
What was shipped (/workspace/output, 11 files, no weights):
train.sh— trains everything from scratch onCORPUS_DIR: a byte-level BPE tokenizer (32,768 merges, trained at run time) and four 41.7M-parameter decoder-only transformers (8 layers, 8 heads, d=512, RoPE, QK-norm, SwiGLU, tied embeddings), seeds 1–4, each with its own 450 s cosine schedule, all inside a wall-clock budget that adapts toTRAIN_BUDGET_SECONDS. Full run: 1839 s of the 3600 s limit.serve.sh/server.py— serves/vocab,/score,/dist,/healthfrom the mean of the four models' next-token distributions, in float32 (TF32 off) so every scoring path agrees.architecture.json,DATA_MANIFEST.md, andlogs/(training log of the shipped run, selection-run table, server start/restart log).
How the recipe was chosen — only on corpus-internal holdouts, never on validation:
- 42M params beat 110M at equal wall clock; dropout 0.3 ≫ 0.2/0.4 ≫ 0.1 ≫ 0.0; attention dropout costs ~35% throughput; ctx 512/1024 tie, 2048 is worse.
- On a 48-document contiguous-block holdout, 450 s (≈1.9e8 tokens) is the optimum: 900 s is 0.03 bpb worse, 1350 s worse still.
- Averaging two independently seeded models measured 1.29168 / 1.29051 alone → 1.26006 averaged, which is why four are trained and served together.
Verification
frontier selfcheckon the final submission: PASSED — every gate (startup 3 s, train.sh exits 0, /vocab, /score, /dist, evaluation deadline 106 s of 3600 s, slowest request 2.0 s of 120 s,/distnormalization 0 nats,/score-vs-/dist0 nats, truncation 0 nats, restart with empty scratch).- Full-budget local run: server ready in 5.6 s, identical answers after a restart, byte-exact UTF-8 round-trip.
One thing worth flagging: an earlier selfcheck run failed the /score-vs-/dist gate (0.025 nats vs 0.02) because bfloat16 kernels pick different reduction shapes for the batched /score path and the single-window /dist path. Switching inference to float32 (with TF32 disabled) brought that to ~1e-6 nats and also made truncation consistency exact.
Measured on the public validation split, the shipped ensemble scores 1.4056 bits/byte (reward 0.263), versus 1.4946 for a single model of the same recipe.