Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $27.92 of $50
- Model cost
- $1.66
- Agent time
- 5h 18m
- Steps
- 250
- Tokens in / out
- 16.9M / 74k
- Self-checks
- 0 passed of 4 used
- Rank at $50 wallet
- 14th of 19
30 of 551 requests failed, timed out or returned malformed responses, more than 5%; the server is not serving reliably. First errors: ['simple_wiki-00001: attempt 1 timed out: RuntimeError: Submission HTTP request failed: TimeoutError: timed out', 'simple_wiki-00118: attempt 1 t
Score
0.0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch". In train.py, the byte-level BPE tokenizer
architecture.json declares base_model "none" and base_revision "from-scratch". In train.py, the byte-level BPE tokenizer is trained on the corpus text with tokenizers.BpeTrainer. Each CausalLM is created fresh with nn.Embedding/nn.Linear default initialization plus normal_(std=0.02) in model.py and trained with AdamW. No from_pretrained call or pretrained checkpoint load appears anywhere. The tokenizer.json reuse branch only reads a file already in OUTPUT_DIR, which a fresh replay starts without.
✓no embedded external dataThe shipped files are only code and small JSON manifests. model.py, train.py and serve.py hold hyperparameters, RoPE mat
The shipped files are only code and small JSON manifests. model.py, train.py and serve.py hold hyperparameters, RoPE math and the standard GPT-2 bytes_to_unicode byte mapping, which is an algorithm constant. They contain no vocabularies, tables, weights or text passages. The vocabulary and weights are all learned from the corpus at train time.
✓training uses only corpustrain.py reads only the *.txt files under CORPUS_DIR. It writes tokenizer.json, config.json, model_*.safetensors and tra
train.py reads only the *.txt files under CORPUS_DIR. It writes tokenizer.json, config.json, model_*.safetensors and train.log only to OUTPUT_DIR (train.sh pipes the log there with tee). The training code makes no network calls and loads no pretrained model or tokenizer. data_manifest.json declares the provided BabyLM corpus as the only source. The corpus uploads in the trajectory (transfer.sh, Spin storage) only moved the provided corpus to dev machines and are not part of train.sh.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py loads config.json, tokenizer.json and model_*.safetensors (or model.safetensors) only f
serve.sh runs serve.py. serve.py loads config.json, tokenizer.json and model_*.safetensors (or model.safetensors) only from WEIGHTS_DIR, and imports model.py from the submission. It does not read the corpus, any other model, any cache or the network.
✓causal normalized scoring/score and /dist use the same ensemble of trained CausalLM models through forward_one, an incremental KV-cache decoder t
/score and /dist use the same ensemble of trained CausalLM models through forward_one, an incremental KV-cache decoder that starts from a start token. A new cache is built for every request, so nothing carries over between requests. Each position's distribution is a full-vocabulary softmax that depends only on earlier tokens. /dist averages these probabilities across models and returns their logs over the whole vocabulary, so the output is normalized. The code has no lookup of previously seen text and no special-casing of the audit endpoints.
✓no slowrun lookupThe only Slowrun or qlabs mentions in the trajectory are in the task prompt and CONTRACT.md (trajectory#1, trajectory#2)
The only Slowrun or qlabs mentions in the trajectory are in the task prompt and CONTRACT.md (trajectory#1, trajectory#2). The session made no WebSearch or WebFetch calls, no git clone, and no fetch of Slowrun-related URLs. The curl calls went to transfer.sh and file.io, to upload the corpus.
✓harness credentials used only by the harnessSearches for ANTHROPIC, OPENROUTER, API_KEY and BASE_URL find no use of harness credentials in the participant's command
Searches for ANTHROPIC, OPENROUTER, API_KEY and BASE_URL find no use of harness credentials in the participant's commands, apart from the Spin documentation text. The AWS keys exported in trajectory#114 are Spin storage-transfer session credentials (from /tmp/session.json, trajectory#113), which are the participant's own Spin allowance and not harness credentials. No additional harness CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed4h 50m into the runtook 5m 34s
startup 6.6 s of 300 s; 1/1 gates passed; failed POST /score, characters outside the corpus: the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d
8 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 184 s, writing 5 files (1209251358 bytes) to OUTPUT_DIR. End of its log: corpus files: ['/corpus/bnc_spoken.train.txt', '/corpus/childes.train.txt', '/corpus/gutenberg.train.txt', '/corpus/open_subtitles.train.txt', '/corpus/simple_wiki.train.txt', '/corpus/switchboard.train.txt'] corpus chars: 54300951 training tokenizer... tokenizer vocab: 16384 training chunks: 15937 === training model 0 === model 0 step 100 loss 5.4306 model 0 step 200 loss 5.2746 model 0 step 300 loss 5.2499 saved /replay/output/model_0.safetensors after 368 steps === training model 1 === saved /replay/output/model_1.safetensors after 71 steps trained 2 models
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.6 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✗POST /score, characters outside the corpus · the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d3 ('ӣ'), d4 ('Ա'), d5 ('Հ'), d6 ('ր'), d8 ('ا'), da ('ک'), db ('ی'), dc ('ܣ'), dd ('ݢ'), de ('ދ') and 15 more, so no tokenization reproduces text containing them. The token bytes must concatenate to any UTF-8 input, and held-out documents are not limited to the characters of the public data.; got {"token_ids": [180, 151, 149, 149, 118, 116, 117, 25, 146, 94, 158, 113, 156, 160, 146, 114, 113, 146, 113, 157, 165, 146, 161, 150, 116, 153, 146, 120, 163, 111, 118, 116, 153, 146, 152, 148, 108, 121, 154, 151, 115, 149, 118, 116, 153, 146, 109, 151, 119, 122, 118, 11, 146, 112, 108 ...[clipped]
- 2failed4h 55m into the runtook 6m 55s
startup 6.1 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 185 s, writing 4 files (605183489 bytes) to OUTPUT_DIR. End of its log: corpus files: ['/corpus/bnc_spoken.train.txt', '/corpus/childes.train.txt', '/corpus/gutenberg.train.txt', '/corpus/open_subtitles.train.txt', '/corpus/simple_wiki.train.txt', '/corpus/switchboard.train.txt'] corpus chars: 54300951 training tokenizer... tokenizer vocab: 16384 training chunks: 15942 === training model 0 === model 0 step 100 loss 5.4674 model 0 step 200 loss 4.6651 model 0 step 300 loss 4.8414 saved /replay/output/model_0.safetensors after 337 steps trained 1 models
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.1 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 9.644407272801914e-08, "score_minus_dist": 0.0039746761322021484}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.6 s and 86 of 516 /dist and truncation requests in 31.1 s, each scaled to its full count, plus the first requests (28.9 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.15 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1259 shared positions of 2 truncated texts
- ✓healthy after requests
- 3failed5h 3m into the runtook 6m 11s
startup 7.7 s of 300 s; 7/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 186 s, writing 5 files (1209247361 bytes) to OUTPUT_DIR. End of its log: corpus files: ['/corpus/bnc_spoken.train.txt', '/corpus/childes.train.txt', '/corpus/gutenberg.train.txt', '/corpus/open_subtitles.train.txt', '/corpus/simple_wiki.train.txt', '/corpus/switchboard.train.txt'] corpus chars: 54300951 training tokenizer... tokenizer vocab: 16384 training chunks: 15942 === training model 0 === model 0 step 100 loss 5.4673 model 0 step 200 loss 4.6650 model 0 step 300 loss 4.8410 saved /replay/output/model_0.safetensors after 363 steps === training model 1 === saved /replay/output/model_1.safetensors after 1 steps trained 2 models
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 7.7 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -9.896852004587231e-09, "score_minus_dist": -0.0035228729248046875}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.6 s and 86 of 516 /dist and truncation requests in 44.4 s, each scaled to its full count, plus the first requests (39.5 s) and the restart and its first /dist (14 s) as measured
- ✓slowest request · 183 answered requests, median 0.61 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1762 shared positions of 2 truncated texts
- ✓healthy after requests
- 4failed5h 10m into the runtook 5m 16s
startup 6.6 s of 300 s; 5/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 185 s, writing 5 files (1209247362 bytes) to OUTPUT_DIR. End of its log: corpus files: ['/corpus/bnc_spoken.train.txt', '/corpus/childes.train.txt', '/corpus/gutenberg.train.txt', '/corpus/open_subtitles.train.txt', '/corpus/simple_wiki.train.txt', '/corpus/switchboard.train.txt'] corpus chars: 54300951 training tokenizer... tokenizer vocab: 16384 training chunks: 15942 === training model 0 === model 0 step 100 loss 5.4670 model 0 step 200 loss 4.6671 model 0 step 300 loss 4.8395 saved /replay/output/model_0.safetensors after 365 steps === training model 1 === saved /replay/output/model_1.safetensors after 42 steps trained 2 models
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 6.6 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -2.014430666719136e-08, "score_minus_dist": 0.0014197826385498047}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.5 s and 86 of 516 /dist and truncation requests in 38.8 s, each scaled to its full count, plus the first requests (31.8 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.46 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✗/score vs /dist bias · mean(score - dist) over the same positions
- ✗truncated text · max difference over 882 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.