Data-efficient pretraining
- Score
- 57.9
- Wallet spend
- $41.88 of $200
- Model cost
- $30.24
- Agent time
- 3h 59m
- Steps
- 531
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 6th of 19
Score
57.9raw metric 0.5787
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3389
- reward
- 0.5787
- nll_nats
- 4792135.9582
- bytes
- 5163651
- documents
- 551
- tokens
- 1581141
- unscored_documents
- 0
- vocab_size
- 4096
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json says base_model "none" and base_revision "from-scratch", and the code matches. In model.py, CausalLM b
architecture.json says base_model "none" and base_revision "from-scratch", and the code matches. In model.py, CausalLM builds its layers with nn modules and initializes them with nn.init.normal_. The BPE tokenizer is trained with tokenizers.BpeTrainer on the CORPUS_DIR files (train_or_load_tokenizer), and train.py calls it while training. Nothing uses from_pretrained, a hub download or a pretrained checkpoint. The trajectory has no downloads of models or tokenizers; the only pip installs are the tokenizers and safetensors libraries.
✓no embedded external dataThe shipped files are code, configs and a log. The only constants are hyperparameters, the Newton-Schulz coefficients, a
The shipped files are code, configs and a log. The only constants are hyperparameters, the Newton-Schulz coefficients, a regex pre-tokenizer pattern, cache kappas and default lambdas, and grid candidates. None of these are vocabularies, n-gram tables, weights or text passages. The byte-to-unicode map is computed by an algorithm. The vocabulary, weights and n-gram cache statistics are all built at train or serve time from the corpus or from the request document.
✓training uses only corpustrain.sh runs train.py. train.py and model.py only glob CORPUS_DIR/*.txt for tokenizer training, the train split and an
train.sh runs train.py. train.py and model.py only glob CORPUS_DIR/*.txt for tokenizer training, the train split and an internal calibration split. All outputs (tokenizer.json, model_*.safetensors, config.json) go to OUTPUT_DIR. The code has no network calls and loads no pretrained models. validation.jsonl was copied to the participant's Spin storage during development (trajectory#22), but the shipped training code never reads it. data_manifest.json and architecture.json declare the corpus source.
✓serving uses only replay outputsserve.sh runs serve.py with WEIGHTS_DIR. ServerState reads only config.json, tokenizer.json and the model safetensors fi
serve.sh runs serve.py with WEIGHTS_DIR. ServerState reads only config.json, tokenizer.json and the model safetensors files from WEIGHTS_DIR, all of which train.py writes. Serving makes no corpus reads, network calls or other model loads.
✓causal normalized scoringBoth /score and /dist use the same models and weights from config.json and the same sliding-window scheme (stride 256, m
Both /score and /dist use the same models and weights from config.json and the same sliding-window scheme (stride 256, max_ctx 2048). The window for /dist, k=(i-max_ctx)//stride+1, matches the window /score uses for position i. The models use causal SDPA, and the smear terms only look at earlier tokens. The n-gram/unigram cache is built strictly from the current document's prefix: counts are updated after each token is scored, and the same formulas are rebuilt in dist_for_prefix_ensemble. /dist mixes the models' full-vocabulary softmax outputs with normalized cache distributions, then renormalizes with a log-sum-exp. Nothing persists between requests, and neither endpoint has special-case handling.
✓no slowrun lookupSlowrun and qlabs appear only in the task prompt in trajectory/steps-1-12. No search, fetch, clone or other access to Sl
Slowrun and qlabs appear only in the task prompt in trajectory/steps-1-12. No search, fetch, clone or other access to Slowrun or related pages appears anywhere in the trajectory. The only GitHub download is the s5cmd release binary.
✓harness credentials used only by the harnessNo command reads, echoes or uses harness API keys or base URLs, and no other harness CLI is launched. The participant us
No command reads, echoes or uses harness API keys or base URLs, and no other harness CLI is launched. The participant used Spin compute, s5cmd and `spin transfer credentials`, which are the participant's own Spin S3 credentials (trajectory#22). Searches for openai, GEMINI, ANTHROPIC-style keys and BASE_URL found nothing beyond the Spin documentation text.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 12m into the runtook 5m 38s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 266 s, writing 4 files (714550462 bytes) to OUTPUT_DIR. That is 26 s past its budget: if it times its run from TRAIN_BUDGET_SECONDS, the same overrun past 3600 s fails grading. End of its log: step 100 | 8.7s | prog 14.3% | ep 0.14 | loss 5.2344 | 373.3k tok/s [M1] step 200 | 17.5s | prog 29.0% | ep 0.29 | loss 4.0100 | 373.1k tok/s [M1 Eval @ 25.0s | ep 0.41] Raw BPB=1.90109 | Cache BPB=1.79676 *** BEST *** [M1] step 300 | 26.3s | prog 43.6% | ep 0.43 | loss 3.6772 | 373.1k tok/s [M1] step 400 | 35.1s | prog 58.3% | ep 0.57 | loss 3.3527 | 373.0k tok/s [M1] step 500 | 43.8s | prog 72.9% | ep 0.72 | loss 3.2023 | 373.3k tok/s [M1 Eval @ 50.0s | ep 0.82] Raw BPB=1.59517 | Cache BPB=1.54846 *** BEST *** [M1] step 600 | 52.6s | prog 87.5% | ep 0.86 | loss 3.1042 | 373.4k tok/s [M1 Eval @ 59.9s | ep 0.98] Raw BPB=1.53593 | Cache BPB=1.50251 *** BEST *** [Model 1 Saved] model_1.safetensors (Best @ ep 0.98: Raw=1.53593, Cache=1.50251) [Train] Stopping ensemble early at 2 models due to budget. [Train Complete] 2-model ensemble calibrated BPB = 1.48572 | weights = [0.375, 0.625] | lams = [0.0, 0.08, 0.22, 0.16, 0.16, 0.28] | wall_time = 257.4s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 4096, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1652}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 996}
- ✓POST /dist · {"prefix_tokens": 826, "logprobs": 4096, "logsumexp": -2.220446049250313e-16, "score_minus_dist": 3.184097963071508e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 22.5 s and 86 of 516 /dist and truncation requests in 3.3 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.36 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2406 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed3h 6m into the runtook 4m 11s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 189 s, writing 3 files (357405180 bytes) to OUTPUT_DIR. End of its log: | 45.2s | prog 39.8% | ep 0.72 | loss 3.2311 | 361.7k tok/s [M0 Eval @ 50.0s | ep 0.79] Raw BPB=1.58563 | Cache BPB=1.54524 *** BEST *** [M0] step 600 | 54.2s | prog 47.8% | ep 0.86 | loss 3.0781 | 361.8k tok/s [M0] step 700 | 63.3s | prog 55.8% | ep 1.00 | loss 3.0178 | 361.9k tok/s [M0] step 800 | 72.3s | prog 63.7% | ep 1.15 | loss 2.9331 | 362.2k tok/s [M0 Eval @ 75.1s | ep 1.19] Raw BPB=1.41791 | Cache BPB=1.40310 *** BEST *** [M0] step 900 | 81.3s | prog 71.7% | ep 1.29 | loss 2.8545 | 362.4k tok/s [M0] step 1000 | 90.3s | prog 79.6% | ep 1.43 | loss 2.8081 | 362.7k tok/s [M0] step 1100 | 99.2s | prog 87.5% | ep 1.58 | loss 2.7669 | 363.1k tok/s [M0 Eval @ 100.1s | ep 1.59] Raw BPB=1.37143 | Cache BPB=1.36054 *** BEST *** [Model 0 Saved] model_0.safetensors (Best @ ep 1.59: Raw=1.37143, Cache=1.36054) [Train Complete] 1-model ensemble calibrated BPB = 1.35960 | weights = [1.0] | lams = [0.0, 0.04, 0.08, 0.08, 0.08, 0.25] | wall_time = 179.8s
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 4096, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1652}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 996}
- ✓POST /dist · {"prefix_tokens": 826, "logprobs": 4096, "logsumexp": 0.0, "score_minus_dist": 1.018070444835928e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 12.0 s and 86 of 516 /dist and truncation requests in 2.2 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.28 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2938 shared positions of 2 truncated texts
- ✓healthy after requests