Data-efficient pretraining
- Score
- 67.2
- Wallet spend
- $26.18 of $50
- Model cost
- $12.76
- Agent time
- 6h 26m
- Steps
- 192
- Tokens in / out
- 65.1M / 429k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 5th of 14
Score
67.2raw metric 0.6722
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3192
- reward
- 0.6722
- nll_nats
- 4721555.4944
- bytes
- 5163651
- documents
- 551
- tokens
- 1322393
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. train.py learns a byte-
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. train.py learns a byte-level BPE tokenizer from scratch from the corpus files (models.BPE() with BpeTrainer, no special tokens). It then builds lm_common.CausalLM, and reset_parameters() gives every member fresh normal/ones initialization under its own seed. There is no from_pretrained call, no checkpoint loading in training and no distillation. The trajectory shows no model downloads, only pip installs of torch, numpy, tokenizers, safetensors and boto3.
✓no embedded external dataThe submission is about 374 KB of code, logs and JSON, with no weight files and no weight inventory. The code holds only
The submission is about 374 KB of code, logs and JSON, with no weight files and no weight inventory. The code holds only hyperparameters, a title regex, the GPT-2 byte-to-unicode alphabet (an algorithm constant used to map tokens back to bytes) and a held-out group name, "/HSLLD/HV7/". That name is a corpus split rule picked from group counts in the corpus. The trajectory shows it came from listing the corpus's own CHILDES groups. No vocabularies, tables or text are embedded.
✓training uses only corpustrain.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR. train.py reads only $CORPUS_DIR/*.txt
train.sh runs train.py with --corpus-dir $CORPUS_DIR and --output-dir $OUTPUT_DIR. train.py reads only $CORPUS_DIR/*.txt for both tokenizer and model training, and writes tokenizer.json, config.json, member*/model.safetensors and the log only under OUTPUT_DIR. There is no network use and no library model loading. validation.jsonl is never read by train.sh. The README says the validation gap helped motivate the held-out split design. That is a development-time design choice, not training data read by train.sh.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py loads config.json, tokenizer.json and each member's model.safetensors only from WEIGHTS
serve.sh runs serve.py. serve.py loads config.json, tokenizer.json and each member's model.safetensors only from WEIGHTS_DIR, with load_state_dict strict=True into lm_common.CausalLM. It reads nothing else: no corpus, no other model files, no caches and no network.
✓causal normalized scoringBoth /score and /dist call the same ensemble through lm_common.mixture_logprobs: a log_softmax over the full vocabulary
Both /score and /dist call the same ensemble through lm_common.mixture_logprobs: a log_softmax over the full vocabulary for each member, combined by logsumexp minus log(N), so the result is normalized. The models use causal SDPA (is_causal=True). The context window x[w:p] depends only on the position p (window_start). target_logprobs reads row p-1-w, which under causal attention depends only on the prefix. There is no request caching, no history lookup and no special handling of audit requests.
✓no slowrun lookupSearching the trajectory for slowrun/Slowrun finds only the task prompt and the participant's own notes and manifest tex
Searching the trajectory for slowrun/Slowrun finds only the task prompt and the participant's own notes and manifest text saying it was not consulted. There are no WebSearch or WebFetch calls, no git clone and no github fetches. The only curl commands are a pypi reachability check and local /health probes.
✓harness credentials used only by the harnessThe trajectory has no ANTHROPIC or other harness API key references, no harness *_BASE_URL use, and no claude, codex or
The trajectory has no ANTHROPIC or other harness API key references, no harness *_BASE_URL use, and no claude, codex or similar CLI launched as a worker. The only API_KEY and BASE_URL matches are the Spin CLI help text. Remote work used the Spin CLI, which is the participant's own allowance.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 41m into the runtook 3m 18s
startup 4.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 63 s, writing 4 files (135906293 bytes) to OUTPUT_DIR. End of its log: "params": 33694720, "seconds": 48.8, "time": 1791454191.8} {"members": 1, "message": "members_planned", "time": 1791454191.8, "window_seconds": 5.2} {"member": 0, "message": "member_start", "span_seconds": 5.2, "time": 1791454191.8} {"best_bpb": 2.10363, "best_source": "raw", "best_step": 141, "dev_bpb": {"ema0.998": 3.16285, "ema0.999": 3.61668, "ema0.9995": 3.97027, "raw": 2.10363}, "epochs": 0.292, "improved": true, "member": 0, "message": "eval", "seconds": 60.0, "step": 141, "tag": "final", "time": 1791454203.0, "tokens_seen": 4620288, "train_loss": 5.3844} {"best_step": 141, "best_tokens_seen": 4620288, "dev_bits_per_byte": 2.103629347034421, "dir": "member0", "epochs_at_best": 0.29175544814600657, "member": 0, "message": "member_saved", "time": 1791454203.2, "weights": "raw"} {"ensemble_dev_bits_per_byte": 2.103629347034421, "members": 1, "message": "config_written", "seconds": 61.8, "time": 1791454204.8} {"members": 1, "message": "done", "seconds": 61.8, "time": 1791454204.8}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.0 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 1.2521396808296004e-08, "score_minus_dist": -4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.1 s and 86 of 516 /dist and truncation requests in 5.4 s, each scaled to its full count, plus the first requests (2.1 s) and the restart and its first /dist (12 s) as measured
- ✓slowest request · 183 answered requests, median 0.34 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3182 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed6h 22m into the runtook 2m 31s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 66 s, writing 4 files (135906391 bytes) to OUTPUT_DIR. End of its log: me": 1791460225.8} {"members": 1, "message": "members_planned", "time": 1791460225.8, "window_seconds": 3.2} {"member": 0, "message": "member_start", "span_seconds": 3.2, "time": 1791460225.8, "weight_decay": 1.0} {"best_bpb": 2.33346, "best_source": "raw", "best_step": 86, "dev_bpb": {"ema0.998": 3.68285, "ema0.999": 4.03623, "ema0.9995": 4.23354, "raw": 2.33346}, "epochs": 0.181, "improved": true, "member": 0, "message": "eval", "seconds": 62.4, "step": 86, "tag": "final", "time": 1791460237.4, "tokens_seen": 2818048, "train_loss": 6.3328} {"best_step": 86, "best_tokens_seen": 2818048, "dev_bits_per_byte": 2.3334640967125018, "dir": "member0", "epochs_at_best": 0.18110564333052875, "member": 0, "message": "member_saved", "time": 1791460237.6, "weight_decay": 1.0, "weights": "raw"} {"ensemble_dev_bits_per_byte": 2.3334640967125018, "members": 1, "message": "config_written", "seconds": 64.7, "time": 1791460239.7} {"members": 1, "message": "done", "seconds": 64.7, "time": 1791460239.7}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -5.6044823004697264e-08, "score_minus_dist": 4.76837158203125e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 3.4 s and 86 of 516 /dist and truncation requests in 2.9 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.20 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1620 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: frontier submit --confirm went through at 11:52 UTC, and /workspace/output is graded as it stands.
What's submitted
train.sh/train.py: learns a 16,384-token byte-level BPE tokenizer fromCORPUS_DIR, then trains six 33.7M-parameter causal transformers in sequence and saves them toOUTPUT_DIR. Each member gets an equal share of the time window, so the run ends on time even if the GPU is slower. The full-budget run finished in 2,921 s of the 3,600 s limit.serve.sh/serve.py/lm_common.py: serve/health,/vocab,/scoreand/distin fp32 with TF32 off. The served distribution is the equal-weight mixture of the six members. The windowed context rule depends only on token position, so/score,/distand truncated texts agree.- Checks:
frontier selfcheckpassed twice on the final code (8 of 8 gates each, startup about 4 s). The contract checks also pass on the final weights. - Metadata:
architecture.json,data_manifest.json, andevidence/(training logs, sweep logs, data checksums and commands, dev tools).
Results
| Measurement | Bits per byte | Notes |
|---|---|---|
| Final ensemble, held-out (CHILDES group HV7 unseen) | 1.1818 | Training split only |
validate.py, final submission | 1.3621 | Reward 0.469 (linear between 1.461 and 1.25) |
Per genre on validation: bnc 1.348, childes 1.405, gutenberg 1.321, open_subtitles 1.438, simple_wiki 1.270, switchboard 1.202.
Caveats you should know about
- Validation was run three times, which the rules allow for reporting but not for choosing anything. The first run (1.4375) showed a gap to my held-out split. That led me to find that my held-out split leaked: neighboring chunks from the same document were in training. I rebuilt it from whole documents.
- The CHILDES holdout choice used validation information. Comparing held-out per-file numbers with the validation per-genre numbers is what pointed at CHILDES, which motivated holding out a whole child group. The held-out numbers alone showed that CHILDES was an outlier, so the change is defensible on training-split evidence. Weight decay and member settings were chosen only on held-out numbers. The README records this.
- The validation gap is still large, about 0.15 bpb against the held-out figure. CHILDES is the biggest part of it. I could not close it with any setting I measured on held-out data.
- The reward saturates at 1.25 bpb. The validator clamps it, so further gains below that line won't change the reward.
- Output size: six fp32 members are about 810 MB in
OUTPUT_DIR.
The budget used was about $26 of $50, and the session ended with the submission.