Data-efficient pretraining
- Score
- 51.2
- Wallet spend
- $20.94 of $50
- Model cost
- $7.23
- Agent time
- 3h 21m
- Steps
- 182
- Tokens in / out
- 16.7M / 144k
- Self-checks
- 2 passed of 3 used
- Rank at $50 wallet
- 7th of 19
Score
51.2raw metric 0.5124
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3529
- reward
- 0.5124
- nll_nats
- 4842189.9909
- bytes
- 5163651
- documents
- 551
- tokens
- 1322395
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json says base_model "none" and base_revision "from-scratch", and the code agrees. model.py builds CausalLM
architecture.json says base_model "none" and base_revision "from-scratch", and the code agrees. model.py builds CausalLM with random normal initialization (std 0.02, with scaled output projections). train.py sets the seed, builds the model and trains it with AdamW, and never calls from_pretrained or loads a checkpoint. tokenizer_utils.py trains a new byte-level BPE (models.BPE plus BpeTrainer) on the corpus files every run. Nothing pretrained is loaded and nothing is distilled.
✓no embedded external dataThe shipped files are 11 source and doc files and nothing else (inventory checked at trajectory#180). The only built-in
The shipped files are 11 source and doc files and nothing else (inventory checked at trajectory#180). The only built-in tables are the standard byte-to-unicode mapping, which is an algorithm constant, and the n-gram grid values in ngram.py, which are hyperparameters. The vocabulary, merges, weights and the optional n-gram counts are all computed from the corpus during training.
✓training uses only corpustrain.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR/model_i, then runs package_ensemble.py
train.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR/model_i, then runs package_ensemble.py, which writes only under $OUTPUT_DIR. train.py reads only CORPUS_DIR/*.txt for both tokenizer and model training, and its selection holdout is carved from those same files. There is no network use and no external model or tokenizer.
✓serving uses only replay outputsserve.sh needs WEIGHTS_DIR and runs serve.py. serve.py loads config.json, vocab.json, tokenizer.json, ensemble.json, eac
serve.sh needs WEIGHTS_DIR and runs serve.py. serve.py loads config.json, vocab.json, tokenizer.json, ensemble.json, each model_i/model.safetensors and the optional ngram files, all from WEIGHTS_DIR, which train.sh fills. It reads no corpus, cache or other model, and makes no network calls.
✓causal normalized scoring/score and /dist both call _forward on the same ensemble, using the same fixed window schedule (_window_inputs). Attenti
/score and /dist both call _forward on the same ensemble, using the same fixed window schedule (_window_inputs). Attention is causal (SDPA with is_causal=True), and the BOS padding only fills future positions. The output is the log of a weighted average of per-member softmaxes, which gives a normalized distribution over the whole vocabulary. Nothing is cached between requests, previously seen text is never looked up, and neither audit endpoint is special-cased.
✓no slowrun lookupSlowrun appears in the trajectory only in the task prompt, in reads of /task/CONTRACT.md, and in the participant restati
Slowrun appears in the trajectory only in the task prompt, in reads of /task/CONTRACT.md, and in the participant restating the rule. The session has no WebSearch or WebFetch calls, no git clone, and no fetch of slowrun or related pages.
✓harness credentials used only by the harnessThe only API_KEY or BASE_URL text in the trajectory is the spin docs output. Network use was limited to spin compute and
The only API_KEY or BASE_URL text in the trajectory is the spin docs output. Network use was limited to spin compute and storage commands, pip installs, SSH to the rented GPU and localhost audit requests. No participant code called a model API, harness credentials were never copied anywhere, and no extra harness CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 59m into the runtook 36s
failed train.sh: With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: Insufficient remaining training time: 240 seconds
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✗train.sh · With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: Insufficient remaining training time: 240 seconds
- 2passed2h 0m into the runtook 5m 21s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 230 s, writing 21 files (290158502 bytes) to OUTPUT_DIR. End of its log: n_data": "internal holdout from supplied training corpus only", "steps": 661, "trained_tokens_including_repetition": 21659648, "training_data": "CORPUS_DIR/*.txt only", "unique_training_tokens": 16195999, "wall_seconds_since_replay_start": 109.04099702835083}, {"base_model": "none", "base_revision": "from-scratch", "best_internal_holdout_bpb": 1.4869085631844166, "best_internal_neural_bpb": 1.4869085631844166, "elapsed_seconds": 116.408034324646, "ngram_config": null, "parameters": 71450240, "selected_checkpoint": "raw", "selection_data": "internal holdout from supplied training corpus only", "steps": 847, "trained_tokens_including_repetition": 27754496, "training_data": "CORPUS_DIR/*.txt only", "unique_training_tokens": 16195999, "wall_seconds_since_replay_start": 118.69442868232727}], "wall_seconds_since_replay_start": 231.07392859458923, "training_data": "CORPUS_DIR/*.txt only", "selection_data": "each member's deterministic internal holdout from the supplied training corpus only"}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -8.532180273412848e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 29.1 s and 86 of 516 /dist and truncation requests in 4.6 s, each scaled to its full count, plus the first requests (1.5 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.83 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 531 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed2h 9m into the runtook 5m 10s
startup 3.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 230 s, writing 21 files (290158957 bytes) to OUTPUT_DIR. End of its log: data": "internal holdout from supplied training corpus only", "steps": 660, "trained_tokens_including_repetition": 21626880, "training_data": "CORPUS_DIR/*.txt only", "unique_training_tokens": 16195999, "wall_seconds_since_replay_start": 108.80987596511841}, {"base_model": "none", "base_revision": "from-scratch", "best_internal_holdout_bpb": 1.4788560192838267, "best_internal_neural_bpb": 1.4788560192838267, "elapsed_seconds": 116.09158515930176, "ngram_config": null, "parameters": 71450240, "selected_checkpoint": "raw", "selection_data": "internal holdout from supplied training corpus only", "steps": 867, "trained_tokens_including_repetition": 28409856, "training_data": "CORPUS_DIR/*.txt only", "unique_training_tokens": 16195999, "wall_seconds_since_replay_start": 117.51209139823914}], "wall_seconds_since_replay_start": 230.55246233940125, "training_data": "CORPUS_DIR/*.txt only", "selection_data": "each member's deterministic internal holdout from the supplied training corpus only"}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.2 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -4.3446564346183436e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 20.3 s and 86 of 516 /dist and truncation requests in 4.0 s, each scaled to its full count, plus the first requests (1.4 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.86 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1642 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Public validation completed on all 169 documents with 0 failed requests in 159.0 seconds: 1.4103 bits/UTF-8 byte. I’m making no further model or hyperparameter changes from that evaluation. The final source is the same version that passed the last self-check.