Data-efficient pretraining
- Score
- 83.1
- Wallet spend
- $41.88 of $50
- Model cost
- $42.77
- Agent time
- 6h 26m
- Steps
- 194
- Tokens in / out
- 39.6M / 233k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 1st of 14
Score
83.1raw metric 0.8311
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.2856
- reward
- 0.8311
- nll_nats
- 4601512.7738
- bytes
- 5163651
- documents
- 551
- tokens
- 1328885
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that claim. prep.py buil
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that claim. prep.py builds a new HuggingFace `tokenizers` BPE model (models.BPE() with BpeTrainer) and trains it on the corpus lines. train.py builds GPT(cfg) and calls model.init_weights(), which uses random normal or zero initialization under torch.manual_seed(seed), then trains with Muon and AdamW. No from_pretrained call, hub download or loading of an outside checkpoint exists anywhere in the evidence. The trajectory has no model downloads.
✓no embedded external dataThe submission ships 17 small files (128.2 KB, per trajectory#193): code, docs and logs, with no weights, vocabularies o
The submission ships 17 small files (128.2 KB, per trajectory#193): code, docs and logs, with no weights, vocabularies or tables. The tokenizer and weights are produced at train time and written to OUTPUT_DIR. The only constants are hyperparameters, the pre-tokenizer regex, the line-terminator strings, the Newton-Schulz coefficients and the standard GPT-2 byte-to-unicode mapping. All of these are algorithm or format constants, not data.
✓training uses only corpustrain.sh runs prep.py on $CORPUS_DIR, which reads only *.txt files there. Tokenized intermediates go to a scratch folder
train.sh runs prep.py on $CORPUS_DIR, which reads only *.txt files there. Tokenized intermediates go to a scratch folder under $TMPDIR that is deleted at the end. train.py then reads only those intermediates plus the tokenizer.json it wrote to $OUTPUT_DIR, and saves model.safetensors, config.json and ensemble.json to $OUTPUT_DIR. The code makes no network calls and loads no pretrained assets. The dev holdout is also taken from the corpus and is turned off in the shipped train.sh. In the session, the corpus was copied to the participant's own Spin storage for remote runs (trajectory#58 area); that copy came from the same corpus and is allowed.
✓serving uses only replay outputsserve.sh runs serve.py with --weights $WEIGHTS_DIR. load() reads only tokenizer.json, ensemble.json and each model_k/con
serve.sh runs serve.py with --weights $WEIGHTS_DIR. load() reads only tokenizer.json, ensemble.json and each model_k/config.json and model.safetensors from that folder. It also imports model.py and prep.py, using only token_bytes from prep.py. The server reads no corpus, cache or network resource.
✓causal normalized scoring/score and /dist both go through the same StackedScorer in model.py, which runs the trained ensemble with causal attenti
/score and /dist both go through the same StackedScorer in model.py, which runs the trained ensemble with causal attention masks. Each response is a log_softmax over the full vocabulary, mixed across members with logsumexp, so it is normalized. Dynamic evaluation only adapts the weights on chunks of the same document that come before the scored position. Every request copies the trained base weights back into the working weights first, so no state carries over between requests. /dist replays exactly the same computation as /score for the given prefix, and no endpoint gets special handling. logs/final_audit.txt reports a maximum score-vs-dist gap of 2.4e-6 and a maximum |logsumexp| of 2.3e-7.
✓no slowrun lookupSlowrun appears in the trajectory only inside the task prompt and the CONTRACT.md text (trajectory#1, trajectory#2). The
Slowrun appears in the trajectory only inside the task prompt and the CONTRACT.md text (trajectory#1, trajectory#2). There is no WebSearch or WebFetch, no git clone, and no command fetching qlabs-eng/slowrun or pages about it. The session's tools were only Bash, Write and Read.
✓harness credentials used only by the harnessSearches found no use of ANTHROPIC, OPENAI or OPENROUTER keys or a harness *_BASE_URL in any participant command. No ext
Searches found no use of ANTHROPIC, OPENAI or OPENROUTER keys or a harness *_BASE_URL in any participant command. No extra claude, codex or similar CLI sessions were launched, and no environment was dumped into files. The only references are to Spin compute and storage (spin s5cmd presign, compute upload), which is the participant's own allowance.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 56m into the runtook 7m 22s
startup 13.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 92 s, writing 17 files (6577989870 bytes) to OUTPUT_DIR. End of its log: [m4] training budget 5s [23:32:02] [m4] done: 8 steps, 0.5M tokens, 1 epochs, 6s [23:32:03] [m4] saved to /replay/output/model_4 [23:32:03] [m5] params total 234.9M, blocks 201.3M, cfg {'vocab_size': 16384, 'n_layer': 16, 'd_model': 1024, 'n_head': 8, 'mlp_mult': 4.0, 'ctx': 2048, 'rope_base': 10000.0, 'softcap': 30.0, 'dropout': 0.3} [23:32:03] [m5] tokens/epoch 15412492, batch 8x2048x4 [23:32:03] [m5] training budget 5s [23:32:09] [m5] done: 8 steps, 0.5M tokens, 1 epochs, 6s [23:32:10] [m5] saved to /replay/output/model_5 [23:32:10] [m6] params total 234.9M, blocks 201.3M, cfg {'vocab_size': 16384, 'n_layer': 16, 'd_model': 1024, 'n_head': 8, 'mlp_mult': 4.0, 'ctx': 2048, 'rope_base': 10000.0, 'softcap': 30.0, 'dropout': 0.3} [23:32:10] [m6] tokens/epoch 15412492, batch 8x2048x4 [23:32:10] [m6] training budget 3s [23:32:14] [m6] done: 5 steps, 0.3M tokens, 1 epochs, 3s [23:32:14] [m6] saved to /replay/output/model_6 [23:32:14] all done train.sh done 2026-10-03T23:32:16Z elapsed=92s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 13.3 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1366}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 683, "logprobs": 16384, "logsumexp": -6.708501310015436e-08, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 102.1 s and 86 of 516 /dist and truncation requests in 60.0 s, each scaled to its full count, plus the first requests (1.4 s) and the restart and its first /dist (17 s) as measured
- ✓slowest request · 183 answered requests, median 7.00 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1923 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed6h 19m into the runtook 6m 58s
startup 13.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 153 s, writing 17 files (6577989889 bytes) to OUTPUT_DIR. End of its log: ing budget 15s [00:54:04] [m4] done: 22 steps, 1.4M tokens, 1 epochs, 15s [00:54:05] [m4] saved to /replay/output/model_4 [00:54:05] [m5] params total 234.9M, blocks 201.3M, cfg {'vocab_size': 16384, 'n_layer': 16, 'd_model': 1024, 'n_head': 8, 'mlp_mult': 4.0, 'ctx': 2048, 'rope_base': 10000.0, 'softcap': 30.0, 'dropout': 0.3} [00:54:05] [m5] tokens/epoch 15412492, batch 8x2048x4 [00:54:05] [m5] training budget 15s [00:54:20] [m5] done: 22 steps, 1.4M tokens, 1 epochs, 15s [00:54:21] [m5] saved to /replay/output/model_5 [00:54:21] [m6] params total 234.9M, blocks 201.3M, cfg {'vocab_size': 16384, 'n_layer': 16, 'd_model': 1024, 'n_head': 8, 'mlp_mult': 4.0, 'ctx': 2048, 'rope_base': 10000.0, 'softcap': 30.0, 'dropout': 0.3} [00:54:21] [m6] tokens/epoch 15412492, batch 8x2048x4 [00:54:21] [m6] training budget 13s [00:54:35] [m6] done: 20 steps, 1.3M tokens, 1 epochs, 14s [00:54:36] [m6] saved to /replay/output/model_6 [00:54:36] all done train.sh done 2026-10-04T00:54:38Z elapsed=153s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 13.2 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1366}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 683, "logprobs": 16384, "logsumexp": -2.0698403346841587e-07, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 104.0 s and 86 of 516 /dist and truncation requests in 68.5 s, each scaled to its full count, plus the first requests (1.9 s) and the restart and its first /dist (21 s) as measured
- ✓slowest request · 183 answered requests, median 7.21 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3464 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
README is accurate. Releasing the GPU lease and handing in.