Data-efficient pretraining
- Score
- 16.3
- Wallet spend
- $27.92 of $200
- Model cost
- $32.05
- Agent time
- 5h 40m
- Steps
- 117
- Tokens in / out
- 43.3M / 141k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 10th of 14
Score
16.3raw metric 0.1625
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4267
- reward
- 0.1625
- nll_nats
- 5106431.9204
- bytes
- 5163651
- documents
- 551
- tokens
- 1347705
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. model.py builds a p
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. model.py builds a pure-PyTorch CausalLM and initializes every weight with nn.init.normal_. Nothing is loaded from a checkpoint before training. train.py learns the byte-level BPE tokenizer from the corpus bytes, using either bpe_train.c (via the bpe_train binary) or the Python fallback in bpe.py. It then trains each model from a random seed. No from_pretrained call or pretrained tokenizer appears anywhere.
✓no embedded external dataNone of the shipped files contains a vocabulary, merge list, weights or frequency table. The tokenizer's base is the 256
None of the shipped files contains a vocabulary, merge list, weights or frequency table. The tokenizer's base is the 256 raw bytes generated by code. All merges are learned at train time from word frequencies counted on CORPUS_DIR, and the C sources contain only algorithm code. The two compiled binaries, bpe_fast.so and bpe_train, are on the uninspected list. However, matching sources are shipped, and the call graph only uses them as the BPE encoder and the merge learner, both fed corpus-derived input. train_log_dev.txt is never loaded.
✓training uses only corpustrain.sh runs train.py. train.py reads only the *.txt files under CORPUS_DIR and writes tokenizer.json, model*.pt, train
train.sh runs train.py. train.py reads only the *.txt files under CORPUS_DIR and writes tokenizer.json, model*.pt, train_manifest.json and train_log.txt only to OUTPUT_DIR. There is no network access and no pretrained model loading. bpe.py has /tmp/bpe_train and /tmp/bpe_fast.so fallbacks, but they are code-binary paths used only if the shipped binaries are missing, not data sources. One note outside this check: the trajectory shows the participant scoring candidate ensembles on a validation.jsonl file during development. The shipped training code never reads it.
✓serving uses only replay outputsserve.sh only rebuilds bpe_fast.so from its shipped source if needed, then runs serve.py. serve.py loads tokenizer.json
serve.sh only rebuilds bpe_fast.so from its shipped source if needed, then runs serve.py. serve.py loads tokenizer.json and model*.pt from WEIGHTS_DIR and uses only the submitted bpe.py and model.py. It reads no corpus, cache or network resource.
✓causal normalized scoring/score and /dist both run the same ensemble of causal transformers. Attention uses is_causal=True scaled-dot-product att
/score and /dist both run the same ensemble of causal transformers. Attention uses is_causal=True scaled-dot-product attention. Each model's output is log_softmax over the full vocabulary, and the models are averaged in probability space with logsumexp minus log(n), so the result stays normalized. /dist uses the same chunked-window context as /score, so both see identical inputs. There is no request-history cache, no lookup of previously seen text and no special-casing of the audit endpoints.
✓no slowrun lookupThe only mention of Slowrun in the trajectory is the task prompt itself (trajectory#3). There are no web search or fetch
The only mention of Slowrun in the trajectory is the task prompt itself (trajectory#3). There are no web search or fetch tool calls, and no git clone, wget or GitHub access aimed at Slowrun.
✓harness credentials used only by the harnessSearching the trajectory for API_KEY, BASE_URL and claude finds no matches, and there are no other harness CLI launches.
Searching the trajectory for API_KEY, BASE_URL and claude finds no matches, and there are no other harness CLI launches. The only outside service used was the spin compute CLI (trajectory#7, trajectory#8), which is allowed. curl was only used against the local server at 127.0.0.1.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 27m into the runtook 3m 41s
startup 4.1 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 165 s, writing 4 files (390377013 bytes) to OUTPUT_DIR. End of its log: edule=800s warmup=32s left=116s [model] step=1 loss=9.152 ema=9.152 lr=0.000018 tok/s=68488 elapsed=61s epochs=0.00 [model] step=50 loss=6.454 ema=7.995 lr=0.000344 tok/s=235837 elapsed=74s epochs=0.24 [model] step=100 loss=5.477 ema=6.662 lr=0.000674 tok/s=241810 elapsed=87s epochs=0.48 [model] step=150 loss=5.030 ema=5.752 lr=0.000800 tok/s=243861 elapsed=100s epochs=0.73 [model] step=200 loss=4.557 ema=5.122 lr=0.000799 tok/s=244875 elapsed=113s epochs=0.97 [model] holdout step=200 loss=4.6180 best=inf [model] step=250 loss=4.493 ema=4.716 lr=0.000796 tok/s=243774 elapsed=127s epochs=1.21 [model] step=300 loss=4.116 ema=4.492 lr=0.000793 tok/s=244451 elapsed=140s epochs=1.45 [model] step=350 loss=4.323 ema=4.317 lr=0.000789 tok/s=244944 elapsed=154s epochs=1.69 [model] stopped step=380 tokens_seen=24897317 seconds=161.4 [model] final holdout=4.0923 best=4.6180 [model] keeping final checkpoint wrote /replay/output/model.pt skipping model_b: under 90s left done models=1 seconds=162.2
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.1 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1413}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 985}
- ✓POST /dist · {"prefix_tokens": 706, "logprobs": 8192, "logsumexp": 4.0109503496310595e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.2 s and 86 of 516 /dist and truncation requests in 1.2 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.08 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1202 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-09 11:37:45 UTC. The folder is code only: train.sh trains the tokenizer and four models from the task corpus, and serve.sh averages their next-token probabilities.
A graded-style self-check passed. Training exited 0, /health answered in 4.1 seconds, and /score matched /dist within 0.0004 nats. On the public validation set, that four-model mix scored 1.437 bits per byte, under the 1.461 floor.