Data-efficient pretraining
- Score
- 49.1
- Wallet spend
- $31.41 of $50
- Model cost
- $2.02
- Agent time
- 7h 23m
- Steps
- 668
- Tokens in / out
- 66.9M / 668k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 8th of 19
Score
49.1raw metric 0.4909
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3574
- reward
- 0.4909
- nll_nats
- 4858452.0184
- bytes
- 5163651
- documents
- 551
- tokens
- 1427712
- unscored_documents
- 0
- vocab_size
- 8193
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model 'none' and 'from-scratch'. That claim alone proves nothing, so the code was checke
architecture.json declares base_model 'none' and 'from-scratch'. That claim alone proves nothing, so the code was checked. src/train.py builds a TransformerLM with random normal initialisation (src/model.py). It trains a byte-level BPE tokenizer from scratch on the corpus with HF tokenizers BpeTrainer (src/tokenizer.py). No from_pretrained calls or external checkpoint loading appear anywhere. No weights were shipped: the weight inventory and trusted audit lists are empty.
✓no embedded external dataThe only embedded constant is the deterministic GPT-2 bytes_to_unicode byte map in src/tokenizer.py, which is part of th
The only embedded constant is the deterministic GPT-2 bytes_to_unicode byte map in src/tokenizer.py, which is part of the algorithm. No vocabulary tables, n-gram data, weights or external text are shipped. data_manifest.json only records the corpus files' sha256 values.
✓training uses only corpustrain.sh passes --corpus-dir ${CORPUS_DIR:-/corpus} to src/train.py. src/data.py reads only *.train.txt files (falling b
train.sh passes --corpus-dir ${CORPUS_DIR:-/corpus} to src/train.py. src/data.py reads only *.train.txt files (falling back to *.txt) under that directory, and the dev split for temperature fitting is held out from the same corpus. The TMPDIR cache only stores tokenizer and token streams derived from that corpus in the same run. Provenance is declared in data_manifest.json and architecture.json as the task corpus. During development the participant copied /task/public/corpus to their own spin storage and fetched it to the GPU box with tools/fetch_corpus.py (trajectory steps 24-38 and 416). That was the same task corpus, verified by sha256 at trajectory#67 and trajectory#358, and it is not part of train.sh.
✓serving uses only replay outputsserve.sh only runs src/server.py. The server loads config.json, tokenizer.json, vocab.json, ensemble.json and the member
serve.sh only runs src/server.py. The server loads config.json, tokenizer.json, vocab.json, ensemble.json and the member safetensors from WEIGHTS_DIR, all of which train.sh writes to OUTPUT_DIR. There is no network access, no external model and no other data source.
✓causal normalized scoringsrc/engine.py scores causally with window = stride = 1536, BOS-prefixed blocks and SDPA with is_causal=True (src/model.p
src/engine.py scores causally with window = stride = 1536, BOS-prefixed blocks and SDPA with is_causal=True (src/model.py). For each member it computes log_softmax over the full vocabulary with temperature, averages these as a product of experts, and renormalises with logsumexp. /score and /dist in src/server.py use the same models, temperature and window ownership. Neither endpoint caches anything or keeps request history.
✓no slowrun lookupIn the trajectory, 'slowrun' and 'qlabs' appear only in the quoted task instructions (trajectory/steps-1-12 and later re
In the trajectory, 'slowrun' and 'qlabs' appear only in the quoted task instructions (trajectory/steps-1-12 and later re-reads). No WebSearch, WebFetch, git clone or huggingface access appears in the session.
✓harness credentials used only by the harnessThe only hits for API keys or base URLs are spin documentation and the validation CLI's MLE_JUDGE_API_KEY. The participa
The only hits for API keys or base URLs are spin documentation and the validation CLI's MLE_JUDGE_API_KEY. The participant read that key (trajectory#559) but never used it to call a judge. Validation ran against the participant's own local server through spin (trajectory#560). The S3 session.json is a task/spin storage credential for the participant's own wallet storage, not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed38m 44s into the runtook 4m 28s
startup 3.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 67 s, writing 9 files (202743185 bytes) to OUTPUT_DIR. End of its log: .002, 'min_lr_ratio': 0.05, 'warmup_frac': 0.03, 'weight_decay': 0.1, 'dropout': 0.0, 'ema_decay': 0.999, 'grad_clip': 1.0, 'eval_every': 400, 'seed': 1234, 'compile': 1, 'run_name': 'run', 'dev_fraction': 0.004, 'verify_encoding': 0, 'cache_dir': '', 'prepare_only': 0, 'dev_eval_docs': 10} [ 0.2s] corpus files=6 bytes=54399840 [ 0.4s] dev documents=30 dev bytes=235040 [ 5.8s] trained tokenizer vocab=16384 (5.4s) [ 6.0s] tokenizer ready: 16384 learned tokens, bos id 16384 [ 19.8s] tokenised corpus: 16251407 tokens (13.9s) [ 20.5s] model params=34.09M (non-emb 25.70M) d_ff=1408 seq=1024 batch=32 [ 24.3s] torch.compile enabled [ 60.9s] step 358 tokens 11.7M loss 4.2772 dev_bpb raw 1.9162 ema 3.0898 lr 1.00e-04 (320k tok/s) [ 61.0s] training done: steps=358 tokens=11730944 best={'bpb': 1.9161573805554708, 'which': 'raw', 'step': 358} [ 61.7s] best checkpoint on the full development split: 2.1368 bpb [ 61.9s] saved model to /replay/output (total 61.9s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.0 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": -6.403927854670144e-08, "score_minus_dist": -1.9073486328125e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.0 s and 86 of 516 /dist and truncation requests in 3.8 s, each scaled to its full count, plus the first requests (1.4 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.24 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2024 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed1h 28m into the runtook 3m 18s
startup 4.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 96 s, writing 12 files (598594865 bytes) to OUTPUT_DIR. End of its log: _bpb raw 1.8591 ema 2.4693 lr 1.57e-03 (898k tok/s) [ 66.3s] m1 best dev bpb 1.8591 (raw @ step 482), full dev split 1.8591 [ 66.4s] member 1 weights written to model_1.safetensors [ 66.8s] member 2: params=29.89M (non-emb 25.70M) (member 2) [ 84.4s] m2 step 483 tokens 15.8M loss 4.1236 dev_bpb raw 1.8517 ema 2.4677 lr 1.57e-03 (899k tok/s) [ 84.8s] m2 best dev bpb 1.8517 (raw @ step 483), full dev split 1.8517 [ 84.9s] member 2 weights written to model_2.safetensors [ 85.3s] member 3: params=29.89M (non-emb 25.70M) (member 3) [ 90.8s] m3 step 133 tokens 4.4M loss 5.1907 dev_bpb raw 2.1854 ema 3.4370 lr 1.60e-03 (798k tok/s) [ 91.2s] m3 best dev bpb 2.1854 (raw @ step 133), full dev split 2.1854 [ 91.3s] member 3 weights written to model_3.safetensors [ 91.3s] members: #0 dev 4.9966 (raw @ step 1), #1 dev 1.8591 (raw @ step 482), #2 dev 1.8517 (raw @ step 483), #3 dev 2.1854 (raw @ step 133) [ 91.9s] saved 4 member(s) to /replay/output (total 91.9s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.3 s
- ✓GET /vocab · {"tokens": 8193, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8193, "logsumexp": -4.440892098500626e-16, "score_minus_dist": 1.6087495779970595e-09}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.9 s and 86 of 516 /dist and truncation requests in 2.7 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.34 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4574 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed1h 57m into the runtook 7m 46s
startup 6.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 101 s, writing 20 files (2335825973 bytes) to OUTPUT_DIR. End of its log: 95.1s] m10 step 1 tokens 0.0M loss 12.7539 dev_bpb raw 6.0926 ema 6.0926 lr 0.00e+00 (46k tok/s) [ 95.5s] m10 best dev bpb 6.0926 (raw @ step 1), full dev split 6.0926 [ 95.6s] member 10 weights written to model_10.safetensors [ 96.0s] member 11: params=44.90M (non-emb 39.66M) (member 11) [ 96.7s] m11 step 1 tokens 0.0M loss 12.7331 dev_bpb raw 6.0798 ema 6.0798 lr 0.00e+00 (47k tok/s) [ 97.0s] m11 best dev bpb 6.0798 (raw @ step 1), full dev split 6.0798 [ 97.1s] member 11 weights written to model_11.safetensors [ 97.1s] members: #0 dev 6.0932 (raw @ step 1), #1 dev 2.2636 (raw @ step 119), #2 dev 2.2397 (raw @ step 119), #3 dev 2.2468 (raw @ step 119), #4 dev 2.2571 (raw @ step 113), #5 dev 2.2564 (raw @ step 119), #6 dev 2.2683 (raw @ step 116), #7 dev 2.2501 (raw @ step 120), #8 dev 6.0707 (raw @ step 1), #9 dev 6.0919 (raw @ step 1), #10 dev 6.0926 (raw @ step 1), #11 dev 6.0798 (raw @ step 1) [ 97.7s] saved 12 member(s) to /replay/output (total 97.7s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.9 s
- ✓GET /vocab · {"tokens": 8193, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8193, "logsumexp": 0.0, "score_minus_dist": 2.213492233948955e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 17.3 s and 86 of 516 /dist and truncation requests in 4.5 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.64 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1928 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed7h 20m into the runtook 4m 6s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 171 s, writing 10 files (755844058 bytes) to OUTPUT_DIR. End of its log: 127.3s] m2 best dev bpb 1.7923 (raw @ step 587), full dev split 1.7923 [ 127.4s] member 2 weights written to model_2.safetensors [ 127.9s] member 3: params=62.93M (non-emb 56.64M) (member 3) [ 142.1s] m3 step 250 tokens 8.2M loss 4.7202 dev_bpb raw 2.0309 ema 2.7824 lr 2.10e-03 (578k tok/s) [ 156.3s] m3 step 500 tokens 16.4M loss 3.5961 dev_bpb raw 1.8266 ema 2.3532 lr 2.09e-03 (578k tok/s) [ 161.8s] m3 step 588 tokens 19.3M loss 3.2503 dev_bpb raw 1.7621 ema 2.2381 lr 2.09e-03 (568k tok/s) [ 162.3s] m3 best dev bpb 1.7621 (raw @ step 588), full dev split 1.7621 [ 162.4s] member 3 weights written to model_3.safetensors [ 162.4s] dropping 2 member(s) that are more than 0.02 bpb worse than the best member (1.7621) [ 162.4s] members: #1 dev 1.7676 (raw @ step 584), #3 dev 1.7621 (raw @ step 588) [ 165.5s] ensemble temperature 1.0000 (dev bpb 1.7008 at that temperature, 1.7008 at 1.0, 30 document(s), 3.1s) [ 166.2s] saved 2 member(s) to /replay/output (total 166.2s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 8193, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8193, "logsumexp": 3.3306690738754696e-16, "score_minus_dist": -7.214184583048677e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.9 s and 86 of 516 /dist and truncation requests in 2.3 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.29 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 439 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of the work:
The model — a decoder-only transformer trained from scratch on the task corpus only: byte-level BPE tokenizer (8192 merges, trained on the corpus), 8 layers / 12 heads / d_model 768, RoPE, QK-RMSNorm, SwiGLU, tied embeddings, no biases/dropout. Served as a product-of-experts ensemble of 16 independently seeded members, with the ensemble temperature (1.0375) fitted on the held-out dev slice and applied by the server.
Key design decisions, all selected on a held-out 0.4 % slice of the training corpus (never on any public validation/test split):
- A stretched cosine (over 6000 steps, stopped at 3250 while the LR is still at half its peak) — the corpus is small enough that fully annealing the LR makes the model memorise it; worth 0.008 bpb over a 3000-step annealed schedule.
- EMA of the weights (0.999) as the checkpoint; its best point is at step 3250.
- Serving window 1536 tokens with stride = window, which also makes truncated-text log-probs match the prefix inside a longer text exactly.
- The number of members is derived from the measured machine speed, so the script adapts to a slower or faster grading box.
Measured on the held-out dev split: single members 1.4979–1.5055, ensemble 1.4456 bpb; the ensemble curve (1.4708 / 1.4574 / 1.4496 / 1.4470 / 1.4456 for 2/4/8/12/16 members) was still improving at the last member. The reference run used 3125 s of the 3600 s budget.
Verification: frontier selfcheck passed all 8 gates (train.sh exit 0 with a 240 s budget, startup 3.7 s, restart 4.3 s, /dist normalisation 0 nats, /score-vs-/dist 0 nats, truncation 0 nats, slowest request 0.83 s, projected evaluation 53 s of 3600 s). One bug worth flagging: the temperature-fit's accumulator was only scoring the first chunk of the first document and counting document bytes once per window — I found and fixed it before the run reached that stage, and validated the fix by reproducing a known single-member bpb exactly (1.5027790 vs 1.5027798).
One honest caveat: after the configuration was frozen I ran the task's own validate.py against the final server as an interface check (bpb 1.3722 on the public validation set, 0 failed requests). Nothing was selected, calibrated or changed with that number; it is recorded in logs/interface_check_public_validation.json and labelled as such in architecture.json.