Data-efficient pretraining
- Score
- 77.5
- Wallet spend
- $18.04 of $50
- Model cost
- $4.63
- Agent time
- 2h 34m
- Steps
- 150
- Tokens in / out
- 18.6M / 151k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 3rd of 14
Score
77.5raw metric 0.7748
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.2975
- reward
- 0.7748
- nll_nats
- 4644003.5281
- bytes
- 5163651
- documents
- 551
- tokens
- 1585919
- unscored_documents
- 0
- vocab_size
- 4096
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds Languag
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds LanguageModel and fills every Linear and Embedding weight with nn.init.normal_. train.py builds this model from Config and trains it with Muon and AdamW. prepare.py trains a new byte-level BPE tokenizer (tokenizers BpeTrainer) on the corpus chunks. Nothing calls from_pretrained or loads a checkpoint during training except the run's own model.pt, averaged_model.pt and similar files when calibrate.py picks one. The trajectory shows pip installs only of library packages (tokenizers, numpy, torch), with no model downloads.
✓no embedded external dataThe shipped files are code, JSON metadata and development logs. No weights, vocabularies or tables are embedded. The tok
The shipped files are code, JSON metadata and development logs. No weights, vocabularies or tables are embedded. The tokenizer, the vocab.json hex list and the n-gram counts (ngram.py fit) are all computed from the corpus at training time. Constants are hyperparameters or algorithm coefficients: the Muon Newton-Schulz polynomial, the 0.75 Kneser-Ney discount and the bucket thresholds. The byte_mapping in prepare.py is the standard procedural GPT-2 byte-to-unicode map, not data. The development logs are never loaded by train.sh or serve.sh. Step 146 of the trajectory checks that no .pt/.npy/.npz files are in the output.
✓training uses only corpustrain.sh runs driver.py, which runs train.py for each ensemble member. prepare.py reads only Path(CORPUS_DIR).rglob('*.t
train.sh runs driver.py, which runs train.py for each ensemble member. prepare.py reads only Path(CORPUS_DIR).rglob('*.txt'). The tokenizer, the neural model and the n-gram model (calibrate.py) are all fit on corpus data. The held-out split used for calibration is a deterministic partition of the corpus training files. All outputs (tokenizer.json, vocab.json, model checkpoints, ngram.npz, inference.json, logs) go to OUTPUT_DIR or its member subdirectories. The code makes no network calls and loads no pretrained models.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py reads only WEIGHTS_DIR: ensemble.json, and per member config.json, inference.json, the
serve.sh runs serve.py. serve.py reads only WEIGHTS_DIR: ensemble.json, and per member config.json, inference.json, the selected weights file, tokenizer.json, vocab.json, data_manifest.json and ngram.npz. All of these are written by train.sh. It does not read the corpus or any other path, and it makes no network calls.
✓causal normalized scoring/score and /dist both go through combined(), which averages probabilities across the same ensemble Predictors. Each one
/score and /dist both go through combined(), which averages probabilities across the same ensemble Predictors. Each one uses the same causal Transformer (SDPA with is_causal=True), the same temperature and the same Kneser-Ney mixture with per-bucket coefficients. inference.py's context_start gives the same window for /score segments and for /dist at each position, so each log-prob depends only on the prefix. /dist applies log_softmax over the full vocab_size, which includes BOS. The interpolated KN distribution sums to 1 because it is built recursively on a normalized unigram, and the convex mixture stays normalized. There is no request caching, no lookup of previously seen text and no special handling of audit requests.
✓no slowrun lookupSearches of the trajectory for 'slowrun' and 'qlabs' match only the task prompt at step 3. The only GitHub fetch is the
Searches of the trajectory for 'slowrun' and 'qlabs' match only the task prompt at step 3. The only GitHub fetch is the cloudflared release binary (trajectory#66 region, steps-46-66), which is used to tunnel the server for validation. No step searches for, fetches or clones Slowrun or anything derived from it.
✓harness credentials used only by the harnessThe trajectory has no references to API_KEY or BASE_URL variables and no printenv. The participant made no model API cal
The trajectory has no references to API_KEY or BASE_URL variables and no printenv. The participant made no model API calls and launched no extra harness CLIs. Network use was limited to Spin compute, pip/uv package installs, a cloudflared download and the frontier selfcheck/submit commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed22m 56s into the runtook 4m 40s
startup 4.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 213 s, writing 11 files (778931491 bytes) to OUTPUT_DIR. End of its log: RPUS_DIR training files only"} {"event": "corpus_internal_calibration", "elapsed": 211.292, "weights": "best_model.pt", "temperature": 1.15, "neural_bpb": 1.5896556592525495, "mixed_bpb": 1.4333034086985061, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "corpus_internal_calibration", "elapsed": 211.564, "weights": "best_model.pt", "temperature": 1.2, "neural_bpb": 1.5892674478145337, "mixed_bpb": 1.436221985951531, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "calibration_complete", "elapsed": 211.565, "seconds": 14.976954460144043, "weights": "averaged_model.pt", "temperature": 1.1, "count_weights": [0.4, 0.4, 0.4, 0.4, 0.4, 0.24, 0.3, 0.31, 0.38, 0.4, 0.34, 0.4, 0.4, 0.4, 0.4, 0.28], "internal_bpb": 1.4303343335462981} {"event": "complete", "elapsed": 211.566, "steps": 4137, "tokens": 135561216, "average_count": 23} {"event": "training_driver_complete", "elapsed": 213.64455151557922, "members": 1}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.4 s
- ✓GET /vocab · {"tokens": 4096, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1645}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 996}
- ✓POST /dist · {"prefix_tokens": 822, "logprobs": 4096, "logsumexp": -6.113467354573743e-08, "score_minus_dist": -5.0769974890840786e-09}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 4.2 s and 86 of 516 /dist and truncation requests in 1.4 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.24 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1703 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed1h 25m into the runtook 5m 5s
startup 8.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 201 s, writing 26 files (3075783027 bytes) to OUTPUT_DIR. End of its log: 9227823692, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "corpus_internal_calibration", "elapsed": 80.885, "weights": "best_model.pt", "temperature": 1.5, "neural_bpb": 1.9289058538523292, "mixed_bpb": 1.6105773696945824, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "calibration_complete", "elapsed": 80.895, "seconds": 25.766038417816162, "weights": "model.pt", "temperature": 1.0, "count_weights": [0.4, 0.4, 0.4, 0.4, 0.4, 0.26, 0.33, 0.35000000000000003, 0.4, 0.4, 0.32, 0.4, 0.4, 0.4, 0.4, 0.21], "internal_bpb": 1.4766626028554624} {"event": "complete", "elapsed": 80.898, "steps": 361, "tokens": 11829248, "average_count": 4, "broad_average_count": 11} {"event": "corpus_internal_ensemble_calibration", "members": 3, "internal_bpb": 1.6912453312080868, "source": "Only held-out blocks from the provided training corpus"} {"event": "training_driver_complete", "elapsed": 201.14745163917542, "members": 3}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 8.9 s
- ✓GET /vocab · {"tokens": 4096, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1645}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 996}
- ✓POST /dist · {"prefix_tokens": 822, "logprobs": 4096, "logsumexp": -1.0656098270445113e-07, "score_minus_dist": 7.052034467491808e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 22.7 s and 86 of 516 /dist and truncation requests in 3.5 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (13 s) as measured
- ✓slowest request · 183 answered requests, median 0.64 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5096 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed2h 29m into the runtook 4m 54s
startup 8.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 196 s, writing 26 files (3075775796 bytes) to OUTPUT_DIR. End of its log: b": 1.6046591657206366, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "corpus_internal_calibration", "elapsed": 66.307, "weights": "broad_averaged_model.pt", "temperature": 1.5, "neural_bpb": 1.9577804937405145, "mixed_bpb": 1.6247907174559075, "source": "Deterministic held-out blocks from CORPUS_DIR training files only"} {"event": "calibration_complete", "elapsed": 66.317, "seconds": 21.48099970817566, "weights": "model.pt", "temperature": 0.95, "count_weights": [0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4, 0.4], "internal_bpb": 1.5415782715530217} {"event": "complete", "elapsed": 66.319, "steps": 240, "tokens": 7864320, "average_count": 2, "broad_average_count": 7} {"event": "corpus_internal_ensemble_calibration", "members": 3, "internal_bpb": 1.7279366065232777, "source": "Only held-out blocks from the provided training corpus"} {"event": "training_driver_complete", "elapsed": 196.36587834358215, "members": 3}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 8.9 s
- ✓GET /vocab · {"tokens": 4096, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1645}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 996}
- ✓POST /dist · {"prefix_tokens": 822, "logprobs": 4096, "logsumexp": 3.37675443162766e-10, "score_minus_dist": -4.097897443600118e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 22.1 s and 86 of 516 /dist and truncation requests in 3.5 s, each scaled to its full count, plus the first requests (1.0 s) and the restart and its first /dist (15 s) as measured
- ✓slowest request · 183 answered requests, median 0.84 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2707 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted code-only in /workspace/output/.
- Full training replay: 58m 21s on one H100.
- Server startup: 8.9 seconds.
- Public validation: 1.304 bits per byte, zero failed requests.
- Final
frontier selfcheck: all 8 gates pass. - Includes
train.sh,serve.sh, source code, architecture metadata, and training evidence—no weights.