Data-efficient pretraining
- Score
- 66.0
- Wallet spend
- $8.15 of $10
- Model cost
- $12.28
- Agent time
- 2h 29m
- Steps
- 288
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 1st of 19
Score
66.0raw metric 0.6598
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3218
- reward
- 0.6598
- nll_nats
- 4730913.5154
- bytes
- 5163651
- documents
- 551
- tokens
- 1428039
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. train.py train
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. train.py trains a new BPE tokenizer with tokenizers.BpeTrainer on the corpus split. Each CausalLM in model.py starts from random weights (_init_weights uses nn.init.normal_) under seeds set in train.py. Nothing in the code calls from_pretrained, loads a checkpoint or distills from another model. The shipped submission is code only, and the trajectory shows weights being produced by train.sh in self-check runs.
✓no embedded external dataThe shipped files contain only code, configs and a log. The only constants are hyperparameters, the Muon Newton-Schulz c
The shipped files contain only code, configs and a log. The only constants are hyperparameters, the Muon Newton-Schulz coefficients, the standard GPT-2 byte-to-unicode mapping, which is generated by an algorithm rather than stored as a table, and the per-file byte and line counts in data_manifest.json, which describe the corpus and are not used by the code. The vocabulary, merges and all weights are learned from the corpus at replay time.
✓training uses only corpustrain.py reads only *.train.txt (or *.txt) files from CORPUS_DIR. If that directory does not exist it falls back to /tas
train.py reads only *.train.txt (or *.txt) files from CORPUS_DIR. If that directory does not exist it falls back to /task/public/corpus, which is the same task corpus. Its outputs (tokenizer.json, *.safetensors, ensemble_config.json) go to OUTPUT_DIR. The only other writes are temporary split copies of the corpus in a mkdtemp scratch directory. The code makes no network calls and loads no pretrained models or tokenizers. The data source is declared in architecture.json and data_manifest.json.
✓serving uses only replay outputsserve.py loads only tokenizer.json, ensemble_config.json and the listed safetensors files from WEIGHTS_DIR, plus model.p
serve.py loads only tokenizer.json, ensemble_config.json and the listed safetensors files from WEIGHTS_DIR, plus model.py from the submission. It does not read the corpus, any other data file or the network.
✓causal normalized scoringBoth /score and /dist use the same weighted ensemble of causal transformers. Attention uses is_causal=True, and each mod
Both /score and /dist use the same weighted ensemble of causal transformers. Attention uses is_causal=True, and each model's output is log-softmaxed with temperature scaling, then the models are mixed with logsumexp. /dist covers the full vocabulary and is renormalized. Long texts use a sliding-window chunking that is the same for both endpoints (split_token_sequence_into_chunks and get_dist_chunk_input), so each log-prob depends only on its prefix. There is no request cache, no lookup of previously seen text and no special handling of audit requests.
✓no slowrun lookupThe only places Slowrun appears in the trajectory are the task prompt and CONTRACT.md. The session used only run_command
The only places Slowrun appears in the trajectory are the task prompt and CONTRACT.md. The session used only run_command, write_to_file, view_file, replace_file_content and manage_task. No step searches for, fetches or clones Slowrun, any GitHub repository or related pages.
✓harness credentials used only by the harnessNo step reads, echoes or uses the harness API keys or base URL, and no step launches another agent CLI. The only BASE_UR
No step reads, echoes or uses the harness API keys or base URL, and no step launches another agent CLI. The only BASE_URL in the participant's code points to the local test server at 127.0.0.1:8000. The other network activity is a pip install of tokenizers and safetensors and the frontier and spin tooling.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 25m into the runtook 2m 19s
startup 3.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 69 s, writing 3 files (183147462 bytes) to OUTPUT_DIR. End of its log: [train] Prepared tokenizer (V=8192) and 17,072,269 train tokens (335 cal docs) in 12.30s [train] Starting model_0 (91,296,793 params, 1771 steps, epochs=6.8, seed=101) [train] Reached time budget deadline at step 25/1771; stopping model_0 early. [train] Finished model_0 (best_ema) in 46.1s -> cal_bpb=4.1481 (T=0.98) [train] Updated ensemble_config.json (1 active models, cal_bpb=4.14814): model_0.safetensors(w=1.000,T=0.98) [train] Remaining time before safety margin is -67.1s (< 253.5s); finishing ensemble with 1 snapshots. [train] All training and calibration complete in 67.1s.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.2 s
- ✓GET /vocab · {"tokens": 8192, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1437}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 0.0, "score_minus_dist": 5.07243633762755e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.5 s and 86 of 516 /dist and truncation requests in 2.5 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.27 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1833 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed2h 23m into the runtook 4m 22s
startup 4.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 204 s, writing 4 files (365753657 bytes) to OUTPUT_DIR. End of its log: [train] Prepared tokenizer (V=8192) and 16,920,734 train tokens (68 buffered cal docs) in 12.61s [train] Starting model_0 (91,296,793 params, 1549 steps, epochs=6.0, seq_len=2048, seed=101) [model_0] step 300/1549 ( 91.5s, 214.9k tok/s, lr=0.951) | loss=3.1308 | ema=1.7044 (best=1.7044) | swa=1.7044 [train] Saved early horizon snapshot model_0_early.safetensors at step 600 (cal_bpb=1.3256, T=1.06) [model_0] step 600/1549 (149.7s, 262.7k tok/s, lr=0.729) | loss=2.7741 | ema=1.3287 (best=1.3287) | swa=1.3287 [train] Reached time budget deadline at step 750/1549; stopping model_0 early. [train] Finished model_0 (best_ema) in 181.0s -> cal_bpb=1.3256 (T=1.06) [train] Updated ensemble_config.json (2 active models, cal_bpb=1.32564): model_0_early.safetensors(w=0.500,T=1.06), model_0.safetensors(w=0.500,T=1.08) [train] Remaining time before safety margin is -7.2s (< 279.5s); finishing ensemble with 2 snapshots. [train] All training and calibration complete in 202.2s.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.4 s
- ✓GET /vocab · {"tokens": 8192, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1435}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 717, "logprobs": 8192, "logsumexp": -2.220446049250313e-16, "score_minus_dist": 2.6170583034001993e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.5 s and 86 of 516 /dist and truncation requests in 2.2 s, each scaled to its full count, plus the first requests (0.4 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.25 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3445 shared positions of 2 truncated texts
- ✓healthy after requests