Data-efficient pretraining
- Score
- 71.1
- Wallet spend
- $34.90 of $50
- Model cost
- $29.08
- Agent time
- 9h 45m
- Steps
- 351
- Tokens in / out
- 36.0M / 337k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 4th of 14
Score
71.1raw metric 0.7111
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3110
- reward
- 0.7111
- nll_nats
- 4692154.4625
- bytes
- 5163651
- documents
- 551
- tokens
- 1585679
- unscored_documents
- 0
- vocab_size
- 4096
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model to "none" and base_revision to "from-scratch", and the code matches. prep.py trains a
architecture.json sets base_model to "none" and base_revision to "from-scratch", and the code matches. prep.py trains a new byte-level BPE tokenizer with tokenizers.BpeTrainer.train_from_iterator, using only the corpus text. model.py builds the GPT and sets its weights randomly in reset_parameters() (normal_/ones_). train.py rebuilds every ensemble member with reset_parameters() and its own seed. No code calls from_pretrained, loads a checkpoint or distills from another model. The only weights loaded anywhere are the members written by train.py, which calibrate.py and serve.py read back. No weight files were shipped.
✓no embedded external dataThe shipped files are code, configuration, README and logs. I found no weights, vocabularies, merge lists, word or n-gra
The shipped files are code, configuration, README and logs. I found no weights, vocabularies, merge lists, word or n-gram tables, or text passages. The constants are hyperparameters, algorithm coefficients (the Newton-Schulz values, the temperature grid) and the standard GPT-2 bytes_to_unicode mapping, which is an algorithm and not learned data. The tokenizer, models and temperature are all computed at run time from the corpus.
✓training uses only corpusrun_train.py hashes the files in CORPUS_DIR for the manifest, then runs prep.py. prep.py reads only *.txt files from COR
run_train.py hashes the files in CORPUS_DIR for the manifest, then runs prep.py. prep.py reads only *.txt files from CORPUS_DIR. Its intermediate token streams go to a temporary scratch directory that is deleted at the end. All other outputs (tokenizer.json, safetensors, configs, ensemble.json, logs) go to OUTPUT_DIR. train.py and calibrate.py read only those prep outputs and the members train.py itself wrote. The code makes no network calls and loads no pretrained tokenizer or model from any library. The data manifest and architecture.json name the provided corpus as the only data source.
✓serving uses only replay outputsserve.sh runs serve.py with WEIGHTS_DIR. serve.py loads tokenizer.json, ensemble.json and each mK.config.json and mK.saf
serve.sh runs serve.py with WEIGHTS_DIR. serve.py loads tokenizer.json, ensemble.json and each mK.config.json and mK.safetensors from WEIGHTS_DIR only, and otherwise imports only the submitted model.py, prep.py and scoring.py. It does not read the corpus, any other model, any cache, or the network. The HTTP server only listens for requests.
✓causal normalized scoringBoth /score and /dist run the same loaded members through the same window plan from scoring.py (plan_windows). Each wind
Both /score and /dist run the same loaded members through the same window plan from scoring.py (plan_windows). Each window starts with BOS followed by the prefix tokens, and attention is causal (is_causal=True). The BOS logit is masked, and outputs are log_softmax over the full vocabulary. Ensemble mixture weights come from each member's summed log-probs on the earlier tokens of the same document, so they depend only on the prefix. /dist renormalizes with logsumexp over the full vocabulary. The code keeps no request history, has no lookup of previously seen text and does not special-case the audit endpoints.
✓no slowrun lookupThe only mentions of Slowrun in the trajectory are the task prompt (trajectory#1) and a grep over the participant's own
The only mentions of Slowrun in the trajectory are the task prompt (trajectory#1) and a grep over the participant's own output files that checks for forbidden strings (trajectory#346). There are no WebSearch or WebFetch calls, no git clone, and no fetch of any Slowrun repository or page about it. The curl calls only check pypi.org headers.
✓harness credentials used only by the harnessSearching the trajectory found no use of ANTHROPIC/OPENAI-style API keys or *_BASE_URL variables, and no extra claude or
Searching the trajectory found no use of ANTHROPIC/OPENAI-style API keys or *_BASE_URL variables, and no extra claude or other harness sessions launched. The credentials the participant handled were Spin transfer (S3) credentials from `spin transfer credentials`, used to move the corpus to a rented Spin machine (trajectory#50, trajectory#51). That is the participant's own Spin allowance, not a harness credential.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed40m 15s into the runtook 1m 6s
startup 12.5 s of 300 s; 1/1 gates passed; failed POST /score: the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text; got {"token_ids": [], "logprobs": []}
7 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 0 s, writing 1 files (8 bytes) to OUTPUT_DIR. End of its log: probe train.sh start Sat Oct 3 22:42:25 UTC 2026 CORPUS_DIR=/corpus HF_DATASETS_OFFLINE=1 HF_HUB_OFFLINE=1 HOME=/replay/home LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64:/usr/local/cuda/lib64 OUTPUT_DIR=/replay/output PATH=/opt/conda/bin:/opt/venv/bin:/usr/local/bin:/usr/bin:/bin PWD=/replay/work PYTHONDONTWRITEBYTECODE=1 REPLAY_START=1791067345 SHLVL=0 TMPDIR=/replay/tmp TRAIN_BUDGET_SECONDS=240 TRANSFORMERS_OFFLINE=1 USER=nobody _=/usr/bin/env CORPUS_DIR=/corpus OUTPUT_DIR=/replay/output REPLAY_START=1791067345 TRAIN_BUDGET_SECONDS=240
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 12.5 s
- ✓GET /vocab · {"tokens": 3, "special": 1}
- ✗POST /score · the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text; got {"token_ids": [], "logprobs": []}
- 2passed7h 43m into the runtook 4m 13s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 187 s, writing 9 files (116700815 bytes) to OUTPUT_DIR. End of its log: 1s ep 1.60 lr 1.19e-03 loss 3.4803 tok/s 417k m0 step 500 t 79s ep 1.78 lr 1.10e-03 loss 3.3583 tok/s 417k m0 step 550 t 86s ep 1.96 lr 9.99e-04 loss 3.2511 tok/s 417k m0 step 600 t 94s ep 2.14 lr 9.02e-04 loss 3.1874 tok/s 417k m0 step 650 t 102s ep 2.32 lr 8.05e-04 loss 3.1023 tok/s 417k m0 step 700 t 110s ep 2.50 lr 7.08e-04 loss 3.0607 tok/s 417k m0 step 750 t 118s ep 2.67 lr 6.11e-04 loss 3.0269 tok/s 417k m0 step 800 t 126s ep 2.85 lr 5.14e-04 loss 2.9766 tok/s 417k m0 step 850 t 134s ep 3.03 lr 4.17e-04 loss 2.9271 tok/s 417k m0 step 900 t 142s ep 3.21 lr 3.20e-04 loss 2.8694 tok/s 417k m0 step 950 t 149s ep 3.39 lr 2.23e-04 loss 2.8667 tok/s 417k m0 step 1000 t 157s ep 3.57 lr 1.26e-04 loss 2.8157 tok/s 417k m0 step 1050 t 165s ep 3.74 lr 2.92e-05 loss 2.7589 tok/s 417k FINAL m0 step 1065 epochs 3.80 time 168s dev_bpb 1.3707 x5:1.368 bnc_spoken:1.541 childes:1.041 gutenberg:1.483 open_subtitles:1.552 simple_wiki:1.545 switchboard:1.461 done total 173s [run_train 187.3s] done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 4096, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1647}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 997}
- ✓POST /dist · {"prefix_tokens": 823, "logprobs": 4096, "logsumexp": -2.220446049250313e-16, "score_minus_dist": 5.7840781053641876e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.1 s and 86 of 516 /dist and truncation requests in 1.7 s, each scaled to its full count, plus the first requests (0.2 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.15 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3843 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed9h 42m into the runtook 4m 3s
startup 3.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 187 s, writing 9 files (116700756 bytes) to OUTPUT_DIR. End of its log: 6k m0 step 450 t 71s ep 1.66 lr 1.20e-03 loss 3.4631 tok/s 416k m0 step 500 t 79s ep 1.84 lr 1.10e-03 loss 3.3738 tok/s 416k m0 step 550 t 87s ep 2.03 lr 1.00e-03 loss 3.3136 tok/s 416k m0 step 600 t 95s ep 2.21 lr 9.07e-04 loss 3.2771 tok/s 416k m0 step 650 t 102s ep 2.40 lr 8.10e-04 loss 3.1866 tok/s 416k m0 step 700 t 110s ep 2.58 lr 7.14e-04 loss 3.1088 tok/s 416k m0 step 750 t 118s ep 2.76 lr 6.17e-04 loss 3.0684 tok/s 416k m0 step 800 t 126s ep 2.95 lr 5.20e-04 loss 3.0186 tok/s 416k m0 step 850 t 134s ep 3.13 lr 4.24e-04 loss 2.9534 tok/s 415k m0 step 900 t 142s ep 3.32 lr 3.27e-04 loss 2.9122 tok/s 415k m0 step 950 t 150s ep 3.50 lr 2.31e-04 loss 2.8883 tok/s 415k m0 step 1000 t 158s ep 3.69 lr 1.34e-04 loss 2.8307 tok/s 415k m0 step 1050 t 166s ep 3.87 lr 3.76e-05 loss 2.7821 tok/s 415k FINAL m0 step 1069 epochs 3.94 time 169s dev_bpb 1.3874 x5:1.387 bnc_spoken:1.557 childes:1.056 gutenberg:1.506 open_subtitles:1.570 simple_wiki:1.576 done total 174s [run_train 187.2s] done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.3 s
- ✓GET /vocab · {"tokens": 4096, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1646}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 998}
- ✓POST /dist · {"prefix_tokens": 823, "logprobs": 4096, "logsumexp": 0.0, "score_minus_dist": -7.657116762871397e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.9 s and 86 of 516 /dist and truncation requests in 1.5 s, each scaled to its full count, plus the first requests (0.1 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.14 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1572 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
I've submitted the code in /workspace/output/. train.sh trains a 4096-token byte-level tokenizer plus an ensemble of 7 small GPTs, and serve.sh serves them.
The final frontier selfcheck passed all 8 gates. It ran train.sh with a 240 s budget, so it only exercised the eager (non-compiled) path.
I ran the compiled path myself on the rented H100 as an unprivileged user, and also with torch.compile forced to fail. Both finished with exit 0, and the failed-compile run printed the fallback to eager mode.
A full-length run on the rented H100 finished in 3385 s with 7 members of about 477 s each. Its held-out score is 1.167 bits-per-byte, and the fitted temperature is 1.053. That held-out slice comes from the training corpus and is optimistic. I ran the public validation file once, after the recipe was final, as a sanity check: it gave 1.2895, which maps to a reward of about 0.81 on the 1.461 to 1.25 scale. No choice was based on it.
Selfcheck notes that the architecture review did not run and can still score the submission zero. architecture.json names the provided corpus as the only data source and lists the training and inference files. The reference-run logs are in logs/reference_run/.