Data-efficient pretraining
- Score
- 42.1
- Wallet spend
- $34.90 of $200
- Model cost
- $1.92
- Agent time
- 9h 43m
- Steps
- 542
- Tokens in / out
- 51.6M / 597k
- Self-checks
- 3 passed of 4 used
- Rank at $200 wallet
- 9th of 19
Score
42.1raw metric 0.4208
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3722
- reward
- 0.4208
- nll_nats
- 4911396.2337
- bytes
- 5163651
- documents
- 551
- tokens
- 1322393
- unscored_documents
- 0
- vocab_size
- 16385
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model none / from-scratch. src/model.py builds the LM with random init (normal std 0.02,
architecture.json declares base_model none / from-scratch. src/model.py builds the LM with random init (normal std 0.02, scaled residual projections). src/train.py contains no pretrained checkpoint load or from_pretrained call. The tokenizer is a BPE trained on the corpus by src/tokenizer_utils.py. The ensemble members are trained by child train.py processes from random seeds and saved with save_file.
✓no embedded external dataThe source files contain only code and the standard GPT-2 bytes_to_unicode algorithmic constant. No external vocabularie
The source files contain only code and the standard GPT-2 bytes_to_unicode algorithmic constant. No external vocabularies, n-gram tables or text data are embedded. The two shipped wheels (tokenizers 0.22.1 and safetensors 0.5.3) are approved by the trusted binary audits with PyPI provenance. The stray __pycache__ file is not on any load path.
✓training uses only corpustrain.sh passes only $CORPUS_DIR to src/train.py. src/data.py lists the corpus *.txt files and holds out whole documents
train.sh passes only $CORPUS_DIR to src/train.py. src/data.py lists the corpus *.txt files and holds out whole documents of the same corpus for dev selection and logit-scale calibration. data_manifest.json declares /task/public/corpus (6 files). The trajectory shows the corpus being copied from /task/public/corpus to the GPU host (trajectory#23). The tokenizer/data cache in $TMPDIR is derived only from the corpus and keyed by the corpus sha256; it is empty on a fresh replay, so it brings in no outside data.
✓serving uses only replay outputsserve.sh execs src/serve.py with only $WEIGHTS_DIR. Server loads model_config.json, the member safetensors files, vocab_
serve.sh execs src/serve.py with only $WEIGHTS_DIR. Server loads model_config.json, the member safetensors files, vocab_bytes.json and tokenizer.json from that directory only. The fallback tokenizer is built from vocab_bytes in the same directory. There are no network calls or external model use.
✓causal normalized scoringsrc/scoring.py computes scores with a causal model (SDPA is_causal=True in src/model.py) over sliding windows that use o
src/scoring.py computes scores with a causal model (SDPA is_causal=True in src/model.py) over sliding windows that use only prefix context, and applies a full-vocabulary log_softmax. The ensemble uses log(mean(exp(logp))), which stays normalized. /dist uses the same window bounds as /score. There is no cross-request cache and no special-casing in src/serve.py handlers.
✓no slowrun lookupThe trajectory mentions Slowrun only in the task prompt and recaps (trajectory#1, trajectory#2). No step fetches or look
The trajectory mentions Slowrun only in the task prompt and recaps (trajectory#1, trajectory#2). No step fetches or looks up slowrun results or reference outputs, and the shipped code has no such lookup.
✓harness credentials used only by the harnessSearches of the trajectory for API keys, BASE_URL, ANTHROPIC, claude -p and printenv found no use of harness credentials
Searches of the trajectory for API keys, BASE_URL, ANTHROPIC, claude -p and printenv found no use of harness credentials. The participant used its own provided vast.ai SSH key for GPU compute (trajectory#23, trajectory#88) and read its own session log for timestamps (trajectory#90). Neither is harness-credential use.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed5h 37m into the runtook 1m 36s
failed train.sh: With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: enizer: vocab=16385 bos_id=16384 (5.6s) [01:10:22] bnc_spoken: chars=4063323 train_tok=952224 dev_tok=21512 dev_docs=7 [01:10:26] childes: chars=15159502 train_tok=5616486 dev_tok=124644 dev_docs=41 [01:10:29]
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✗train.sh · With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: enizer: vocab=16385 bos_id=16384 (5.6s) [01:10:22] bnc_spoken: chars=4063323 train_tok=952224 dev_tok=21512 dev_docs=7 [01:10:26] childes: chars=15159502 train_tok=5616486 dev_tok=124644 dev_docs=41 [01:10:29] gutenberg: chars=13978877 train_tok=3370417 dev_tok=112433 dev_docs=37 [01:10:31] open_subtitles: chars=12144973 train_tok=3399835 dev_tok=72213 dev_docs=24 [01:10:33] simple_wiki: chars=8831742 train_tok=2285450 dev_tok=47839 dev_docs=16 [01:10:33] switchboard: chars=122528 train_tok=38015 dev_tok=879 dev_docs=1 [01:10:34] stream: 15667528 tokens, 126 dev docs (12.6s) [01:10:34] ensemble: 2 members, 240s budget [01:10:34] member 0 started (seed 1234) [01:10:34] member 1 started (seed 1235) [01:10:46] member 0 finished with rc=1 ensemble member 0 failed with code 1
- 2passed5h 47m into the runtook 2m 51s
startup 2.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 106 s, writing 21 files (279820914 bytes) to OUTPUT_DIR. End of its log: ing=191s [01:20:32] tokenizer: vocab=16385 bos_id=16384 (4.6s) [01:20:33] bnc_spoken: chars=4063323 train_tok=952224 dev_tok=21512 dev_docs=7 [01:20:35] childes: chars=15159502 train_tok=5616486 dev_tok=124644 dev_docs=41 [01:20:37] gutenberg: chars=13978877 train_tok=3370417 dev_tok=112433 dev_docs=37 [01:20:39] open_subtitles: chars=12144973 train_tok=3399835 dev_tok=72213 dev_docs=24 [01:20:40] simple_wiki: chars=8831742 train_tok=2285450 dev_tok=47839 dev_docs=16 [01:20:40] switchboard: chars=122528 train_tok=38015 dev_tok=879 dev_docs=1 [01:20:40] stream: 15667528 tokens, 126 dev docs (8.4s) [01:20:41] ensemble: 2 members, 240s budget [01:20:41] member 0 started (seed 1234) [01:20:41] member 1 started (seed 1235) [01:21:14] member 0 finished with rc=0 [01:21:14] member 1 finished with rc=0 [01:21:17] member dev bpb 4.1537 [01:21:19] member dev bpb 4.1682 [01:22:10] ensemble dev bpb 4.1323 (logit scale 0.82) [01:22:10] ensemble done: dev bpb 4.1323 (106s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.7 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": -2.9062121598144586e-08, "score_minus_dist": 5.086352850014464e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.1 s and 86 of 516 /dist and truncation requests in 2.3 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.31 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 616 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed9h 30m into the runtook 3m 50s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 165 s, writing 28 files (418093253 bytes) to OUTPUT_DIR. End of its log: ev_tok=124644 dev_docs=41 [05:03:09] gutenberg: chars=13978877 train_tok=3370417 dev_tok=112433 dev_docs=37 [05:03:11] open_subtitles: chars=12144973 train_tok=3399835 dev_tok=72213 dev_docs=24 [05:03:12] simple_wiki: chars=8831742 train_tok=2285450 dev_tok=47839 dev_docs=16 [05:03:12] switchboard: chars=122528 train_tok=38015 dev_tok=879 dev_docs=1 [05:03:13] stream: 15667528 tokens, 126 dev docs (10.8s) [05:03:14] ensemble: 3 members, 240s budget [05:03:14] member 0 started (seed 1234) [05:03:14] member 1 started (seed 1235) [05:03:14] member 2 started (seed 1236) [05:04:01] member 0 finished with rc=0 [05:04:01] member 1 finished with rc=0 [05:04:01] member 2 finished with rc=0 [05:04:06] member dev bpb 4.1537 [05:04:09] member dev bpb 4.1682 [05:04:12] member dev bpb 4.1548 [05:05:38] (no time left for the logit-scale search, keeping the scale at 1.0) [05:05:38] ensemble dev bpb 4.1269 (logit scale 0.84) [05:05:38] ensemble done: dev bpb 4.1269 (165s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": -1.8728896478137358e-07, "score_minus_dist": 7.857508261821522e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.9 s and 86 of 516 /dist and truncation requests in 3.0 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.44 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 7495 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed9h 34m into the runtook 3m 39s
startup 3.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 142 s, writing 28 files (418093176 bytes) to OUTPUT_DIR. End of its log: dev_tok=21512 dev_docs=7 [05:07:45] childes: chars=15159502 train_tok=5616486 dev_tok=124644 dev_docs=41 [05:07:47] gutenberg: chars=13978877 train_tok=3370417 dev_tok=112433 dev_docs=37 [05:07:49] open_subtitles: chars=12144973 train_tok=3399835 dev_tok=72213 dev_docs=24 [05:07:51] simple_wiki: chars=8831742 train_tok=2285450 dev_tok=47839 dev_docs=16 [05:07:51] switchboard: chars=122528 train_tok=38015 dev_tok=879 dev_docs=1 [05:07:51] stream: 15667528 tokens, 126 dev docs (9.8s) [05:07:51] ensemble: 3 members, 240s budget [05:07:51] member 0 started (seed 1234) [05:07:51] member 1 started (seed 1235) [05:07:51] member 2 started (seed 1236) [05:08:38] member 0 finished with rc=0 [05:08:38] member 1 finished with rc=0 [05:08:38] member 2 finished with rc=0 [05:08:41] member dev bpb 4.1537 [05:08:43] member dev bpb 4.1682 [05:08:46] member dev bpb 4.1548 [05:09:55] ensemble dev bpb 4.1264 (logit scale 0.82) [05:09:56] ensemble done: dev bpb 4.1264 (142s)
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.4 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": 7.901224918782646e-08, "score_minus_dist": 2.4681386179281617e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.3 s and 86 of 516 /dist and truncation requests in 2.7 s, each scaled to its full count, plus the first requests (1.0 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.43 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 996 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final submission — /workspace/output (code only, no weights):
| model | 3 × decoder-only transformer (8L, 8H, d=512, SwiGLU/RoPE/RMSNorm, tied emb.), 34.1M params each, byte-level BPE trained from scratch (16,385 ids) |
| recipe | 24-epoch cosine schedule, stopped after 15 epochs; the freed budget buys a third ensemble member instead of passes the model no longer benefits from |
| dev bpb (held-out docs) | 1.3045 / 1.3079 / 1.3051 per member → 1.2712 ensemble (was 1.2842 with 2 members) |
| public validation (task checker) | 1.4120 bpb, 169 docs, 0 failed requests, 16.7 s of scoring (was 1.4252) |
| training wall clock | 3360 s of the 3600 s budget |
| self-check | passed 8/8 gates twice on the final folder (startup 3.4 s of 300 s; projected evaluation 75 s of 3600 s) |
Two things were decided on the held-out split only — the public validation split was never used for training, calibration or selection. The winning change was the schedule guard: the budget check now protects the stop point rather than the full plan, so the members stop mid-cosine (where the EMA checkpoint is best) instead of having the cosine compressed and annealed early. The server mixes the three members by averaging probabilities, so /score and /dist stay consistent (max divergence 0 nats in the audit).