Data-efficient pretraining
- Score
- 80.5
- Wallet spend
- $108.19 of $200
- Model cost
- $25.94
- Agent time
- 10h 31m
- Steps
- 332
- Tokens in / out
- 31.0M / 288k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 3rd of 14
Score
80.5raw metric 0.8048
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.2912
- reward
- 0.8048
- nll_nats
- 4621403.5523
- bytes
- 5163651
- documents
- 551
- tokens
- 1427905
- unscored_documents
- 0
- vocab_size
- 8256
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch". data.py trains a fresh byte-level BPE (mo
architecture.json declares base_model "none" and base_revision "from-scratch". data.py trains a fresh byte-level BPE (models.BPE() with BpeTrainer on corpus text). lm.GPT.init_weights randomly initializes all weights (normal or zero), and train.py builds models with build_model/init_weights and trains them with Muon and AdamW. Nothing is loaded from a pretrained model or tokenizer. BRANCH_LOAD and CACHE_DIR are dev-only options that default to empty and are not set by train.sh. The trajectory's final run (trajectory#317) launches train.sh on the corpus.
✓no embedded external dataThe shipped code (train.py, lm.py, data.py, serve.py) contains only hyperparameters, algorithm constants (Muon Newton-Sc
The shipped code (train.py, lm.py, data.py, serve.py) contains only hyperparameters, algorithm constants (Muon Newton-Schulz coefficients, the RoPE base) and the standard GPT-2 bytes_to_unicode mapping, which is computed by formula and is not data. It contains no vocabularies, word lists, weights or text. data_manifest.json declares no other data.
✓training uses only corpustrain.sh runs train.py, which reads only $CORPUS_DIR/*.txt through data.read_files/load_split and writes the tokenizer,
train.sh runs train.py, which reads only $CORPUS_DIR/*.txt through data.read_files/load_split and writes the tokenizer, model*.safetensors, calib.json and logs to $OUTPUT_DIR. The cache and branch paths are disabled by default. The code makes no network calls. During development the corpus was tarred from /task/public and uploaded to the Spin machines (trajectory#16-33 region). The validation file was packed in the same tarball but is not under the corpus glob and is not used by train.py.
✓serving uses only replay outputsserve.py loads tokenizer.json, model*.safetensors (with their .json configs, via lm.load_model) and calib.json from WEIG
serve.py loads tokenizer.json, model*.safetensors (with their .json configs, via lm.load_model) and calib.json from WEIGHTS_DIR only. It reads no corpus, other model, cache or network resource.
✓causal normalized scoringBoth /score (lm.doc_logprobs) and /dist (lm.next_dist) use the same loaded ensemble, the same masked_logprobs with the s
Both /score (lm.doc_logprobs) and /dist (lm.next_dist) use the same loaded ensemble, the same masked_logprobs with the same temperature, and a consistent window_start rule, so each token's context is only the BOS plus the preceding tokens. Attention is causal (is_causal=True). /dist applies log_softmax over the full vocabulary, with padded and BOS entries masked to -inf and clamped to -1e9, so it is normalized. There is no request caching or special-casing of the audit endpoints, and each request is computed independently under a lock.
✓no slowrun lookupEvery Slowrun or qlabs match in the trajectory comes from the task prompt or from reading /task/CONTRACT.md (e.g. trajec
Every Slowrun or qlabs match in the trajectory comes from the task prompt or from reading /task/CONTRACT.md (e.g. trajectory#108, trajectory#141), or from the participant's own compliance notes. The trajectory contains no web search, fetch, git clone or download targeting Slowrun.
✓harness credentials used only by the harnessThe trajectory contains no use of ANTHROPIC/OPENAI API keys or base URLs, no anthropic SDK calls and no extra claude/cod
The trajectory contains no use of ANTHROPIC/OPENAI API keys or base URLs, no anthropic SDK calls and no extra claude/codex sessions. trajectory#8 only prints the first five characters of HF_TOKEN, which is not a harness credential. All remote work went through the spin CLI.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 28m into the runtook 2m 45s
startup 3.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 115 s, writing 6 files (357393127 bytes) to OUTPUT_DIR. End of its log: docs 143 bytes 1263677 tokens 406360 [ 20.4s] model 0: 89.2M params, cfg {'vocab': 8256, 'n_layer': 16, 'n_embd': 640, 'n_head': 5, 'mlp_mult': 4, 'ctx': 2048, 'drop': 0.5, 'emb_drop': 0.2, 'attn_drop': 0.1, 'tie': False, 'softcap': 0.0, 'rope_base': 10000.0, 'act': 'relu2', 'label_smooth': 0.0, 'loops': 1}, budget 20s [ 67.1s] compile/warmup done in 47s [ 81.9s] step 100 ep 0.39 prog 0.730 lr 0.541 loss 4.1319 tok/s 442320 [ 87.1s] finished model 0: steps 134, epochs 0.5, train time 20s [ 92.7s] model 0 final holdout bpb 1.7141 {'bnc_spoken': 1.7958, 'childes': 1.4063, 'gutenberg': 1.7496, 'open_subtitles': 1.8319, 'simple_wiki': 1.9779} [ 98.3s] model 0 best temperature 1.1 -> holdout bpb 1.7076 [ 98.9s] model 0 saved; elapsed since replay start 102s [ 98.9s] only 102s left before the deadline; stopping with 1 models [ 109.2s] ensemble of 1: holdout bpb 1.7141; best temperature 1.1 -> 1.7076 [ 109.2s] done: 1 models, temperature 1.1, elapsed since replay start 113s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.3 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1438}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 719, "logprobs": 8256, "logsumexp": 1.4157279171200798e-07, "score_minus_dist": -4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.8 s and 86 of 516 /dist and truncation requests in 1.5 s, each scaled to its full count, plus the first requests (0.2 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 183 answered requests, median 0.19 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4107 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed8h 36m into the runtook 4m 36s
startup 5.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 189 s, writing 6 files (357393168 bytes) to OUTPUT_DIR. End of its log: docs 143 bytes 1263677 tokens 406360 [ 32.9s] model 0: 89.2M params, cfg {'vocab': 8256, 'n_layer': 16, 'n_embd': 640, 'n_head': 5, 'mlp_mult': 4, 'ctx': 2048, 'drop': 0.5, 'emb_drop': 0.2, 'attn_drop': 0.1, 'tie': False, 'softcap': 0.0, 'rope_base': 10000.0, 'act': 'relu2', 'label_smooth': 0.0, 'loops': 1}, budget 20s [ 123.9s] compile/warmup done in 91s [ 139.5s] step 100 ep 0.39 prog 0.775 lr 0.449 loss 4.2006 tok/s 418381 [ 144.0s] finished model 0: steps 128, epochs 0.5, train time 20s [ 153.0s] model 0 final holdout bpb 1.7437 {'bnc_spoken': 1.8113, 'childes': 1.4457, 'gutenberg': 1.7708, 'open_subtitles': 1.8535, 'simple_wiki': 2.0226} [ 162.5s] model 0 best temperature 1.0 -> holdout bpb 1.7437 [ 163.5s] model 0 saved; elapsed since replay start 168s [ 163.5s] only 36s left before the deadline; stopping with 1 models [ 181.5s] ensemble of 1: holdout bpb 1.7437; best temperature 1.0 -> 1.7437 [ 181.5s] done: 1 models, temperature 1.0, elapsed since replay start 186s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.5 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1438}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 719, "logprobs": 8256, "logsumexp": 3.9546307339932696e-08, "score_minus_dist": 1.430511474609375e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.9 s and 86 of 516 /dist and truncation requests in 2.5 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (13 s) as measured
- ✓slowest request · 183 answered requests, median 0.26 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4463 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed10h 28m into the runtook 2m 54s
startup 3.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 119 s, writing 6 files (357393171 bytes) to OUTPUT_DIR. End of its log: docs 143 bytes 1263677 tokens 406360 [ 24.0s] model 0: 89.2M params, cfg {'vocab': 8256, 'n_layer': 16, 'n_embd': 640, 'n_head': 5, 'mlp_mult': 4, 'ctx': 2048, 'drop': 0.4, 'emb_drop': 0.2, 'attn_drop': 0.1, 'tie': False, 'softcap': 0.0, 'rope_base': 10000.0, 'act': 'relu2', 'label_smooth': 0.0, 'loops': 1}, budget 20s [ 70.5s] compile/warmup done in 46s [ 84.5s] step 100 ep 0.39 prog 0.692 lr 0.615 loss 4.1769 tok/s 468505 [ 90.6s] finished model 0: steps 143, epochs 0.6, train time 20s [ 96.2s] model 0 final holdout bpb 1.7005 {'bnc_spoken': 1.7769, 'childes': 1.4086, 'gutenberg': 1.7296, 'open_subtitles': 1.8068, 'simple_wiki': 1.9653} [ 101.9s] model 0 best temperature 1.0 -> holdout bpb 1.7005 [ 102.5s] model 0 saved; elapsed since replay start 105s [ 102.5s] only 99s left before the deadline; stopping with 1 models [ 113.9s] ensemble of 1: holdout bpb 1.7005; best temperature 1.0 -> 1.7005 [ 113.9s] done: 1 models, temperature 1.0, elapsed since replay start 117s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.6 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1438}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 719, "logprobs": 8256, "logsumexp": 5.22592473739536e-08, "score_minus_dist": -1.430511474609375e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.2 s and 86 of 516 /dist and truncation requests in 1.7 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.26 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5108 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
I submitted /workspace/output, and the final frontier selfcheck passed all its gates on the same files. The score from that submission isn't known yet. The one measurement I have is public validation on the final models: 1.2790 bits per byte, reward about 0.863. That is a measurement only; nothing was tuned on it.
- Model: a byte-level BPE tokenizer with an 8192-token vocabulary and an ensemble of five transformers, about 89M parameters each.
train.shtrains everything from scratch on the corpus alone. It took about 3360 s in my full run, inside the 3600 s limit. - Serving:
serve.shserves/vocab,/scoreand/distfrom the trained files. A softmax temperature fitted on a 2.5% holdout cut from the training corpus is applied in both/scoreand/dist. - Changes this session:
- Lower learning rates for the output layer and higher ones for the embeddings.
- Dropout reduced from 0.5 to 0.4.
- Faster attention during training.
- These moved the 5-model ensemble's holdout score from 1.1875 to 1.1715 bits per byte, and public validation from 1.2952 to 1.2790.
- Experiments that didn't help: BPE dropout, lower dropout (0.3), the output-layer initialisation scale, Muon learning rate, and weight decay.
- Compliance: I did not look up NanoGPT Slowrun, and I did not use the validation split for training, calibration or choosing settings.
architecture.json,data_manifest.jsonandlogs/(training log, per-model results,calib.json) are included. - Risk: the selfcheck does not run the architecture review, which the tool says can still score a submission zero.