Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $48.86 of $200
- Model cost
- $4.25
- Agent time
- 9h 14m
- Steps
- 378
- Tokens in / out
- 38.6M / 110k
- Self-checks
- 2 passed of 4 used
- Rank at $200 wallet
- 15th of 19
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4979
- reward
- 0
- nll_nats
- 5361150.1799
- bytes
- 5163651
- documents
- 551
- tokens
- 1256035
- unscored_documents
- 0
- vocab_size
- 32768
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py trains a n
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py trains a new byte-level BPE tokenizer from scratch with tokenizers' BpeTrainer on the corpus files. It builds GPT models from model.py with random normal(0, 0.02) initialization and trains them with AdamW. The code makes no from_pretrained calls and loads no checkpoints, and the trajectory shows no model downloads.
✓no embedded external dataThe shipped files are only code: train.sh, train.py, model.py, serve.sh, serve.py and architecture.json. The only consta
The shipped files are only code: train.sh, train.py, model.py, serve.sh, serve.py and architecture.json. The only constants are hyperparameters, RoPE frequencies and the standard GPT-2 bytes_to_unicode byte mapping, which is an algorithmic constant. No vocabularies, tables, weights or encoded data are embedded. The tokenizer and weights are produced at replay time.
✓training uses only corpustrain.py reads only $CORPUS_DIR/*.train.txt. It writes tokenizer.json, data_manifest.json, model_*.pt and train.log only
train.py reads only $CORPUS_DIR/*.train.txt. It writes tokenizer.json, data_manifest.json, model_*.pt and train.log only to $OUTPUT_DIR. The code makes no network calls and loads no pretrained resources. The data manifest names the BabyLM 10M-word corpus from CORPUS_DIR as its source.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py loads model*.pt and tokenizer.json only from $WEIGHTS_DIR, plus the submitted model.py.
serve.sh runs serve.py. serve.py loads model*.pt and tokenizer.json only from $WEIGHTS_DIR, plus the submitted model.py. It reads no other data, caches or network resources.
✓causal normalized scoring/score and /dist both call run_model, which averages the logits of the same model ensemble over [bos] + tokens. Attentio
/score and /dist both call run_model, which averages the logits of the same model ensemble over [bos] + tokens. Attention is causal (scaled_dot_product_attention with is_causal=True, plus RoPE), so each log-prob depends only on the prefix. /dist applies log_softmax over the full vocabulary. Each request is computed fresh, with no history cache, lookup or special-casing of the audit endpoints.
✓no slowrun lookupSlowrun and qlabs appear in the trajectory only in the task prompt and CONTRACT.md (trajectory#1, trajectory#2). There a
Slowrun and qlabs appear in the trajectory only in the task prompt and CONTRACT.md (trajectory#1, trajectory#2). There are no web searches, and the only GitHub fetches are the s5cmd release binary (a file-transfer tool), which is unrelated to Slowrun.
✓harness credentials used only by the harnessSearches of the trajectory for ANTHROPIC, OPENROUTER, API_KEY usage, BASE_URL usage, printenv and claude found nothing b
Searches of the trajectory for ANTHROPIC, OPENROUTER, API_KEY usage, BASE_URL usage, printenv and claude found nothing beyond Spin documentation text. Compute and transfers went through the Spin CLI only (for example trajectory#372 and trajectory#375). No harness credentials were used or copied, and no extra harness sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed8h 49m into the runtook 5m 44s
train.sh, given TRAIN_BUDGET_SECONDS=240, was still running when the self-check stopped it after 311 s, so serve.sh, the restart, the speed gates and the audits were not exercised; a train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them.
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, still running without error on NVIDIA H100 80GB HBM3 after 311 s; stopped there, so serve.sh, the startup deadline, the evaluation deadlines and the audits were not exercised. A train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them. End of its log: del 0 epoch 5 step 2200 loss 2.9815 time 102.2 model 0 epoch 5 step 2400 loss 3.0218 time 109.7 model 0 epoch 6 step 2600 loss 3.4320 time 117.2 model 0 epoch 6 step 2800 loss 3.1531 time 124.7 model 0 epoch 7 step 3000 loss 2.9771 time 132.2 model 0 epoch 7 step 3200 loss 3.1285 time 139.7 model 0 epoch 8 step 3400 loss 3.0646 time 147.2 model 0 done steps 3475 === model 1 === model 1 epoch 1 step 200 loss 4.3790 time 8.0 model 1 epoch 1 step 400 loss 4.2989 time 15.5 model 1 epoch 2 step 600 loss 3.7058 time 23.0 model 1 epoch 2 step 800 loss 3.9301 time 30.5 model 1 epoch 3 step 1000 loss 3.9645 time 38.0 model 1 epoch 3 step 1200 loss 3.8504 time 45.5 model 1 epoch 3 step 1400 loss 3.2016 time 53.0 model 1 epoch 4 step 1600 loss 3.4970 time 60.5 model 1 epoch 4 step 1800 loss 3.4892 time 68.0 model 1 epoch 5 step 2000 loss 3.3895 time 75.5 model 1 epoch 5 step 2200 loss 3.0998 time 83.0 model 1 epoch 5 step 2400 loss 3.0375 time 90.5 model 1 epoch 6 step 2600 loss 3.2400 time 98.0
- 2failed8h 55m into the runtook 4m 4s
startup 2.6 s of 300 s; 1/1 gates passed; failed POST /score, characters outside the corpus: the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d
8 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 200 s, writing 4 files (119772154 bytes) to OUTPUT_DIR. End of its log: n.txt', '/corpus/switchboard.train.txt'] tokenizer vocab 32768 bos_id 0 === model 0 === model 0 epoch 1 step 200 loss 4.6052 time 23.1 model 0 epoch 1 step 400 loss 4.4343 time 30.5 model 0 epoch 2 step 600 loss 3.7457 time 37.9 model 0 epoch 2 step 800 loss 4.0129 time 45.4 model 0 epoch 3 step 1000 loss 3.6891 time 52.8 model 0 epoch 3 step 1200 loss 3.4295 time 60.2 model 0 epoch 3 step 1400 loss 3.8600 time 67.6 model 0 epoch 4 step 1600 loss 3.2701 time 75.1 model 0 epoch 4 step 1800 loss 3.6460 time 82.5 model 0 epoch 5 step 2000 loss 3.5515 time 89.9 model 0 epoch 5 step 2200 loss 2.9810 time 97.4 model 0 epoch 5 step 2400 loss 3.0229 time 104.8 model 0 epoch 6 step 2600 loss 3.4326 time 112.3 model 0 epoch 6 step 2800 loss 3.1524 time 119.7 model 0 epoch 7 step 3000 loss 2.9751 time 127.2 model 0 epoch 7 step 3200 loss 3.1277 time 134.6 model 0 epoch 8 step 3400 loss 3.0677 time 142.0 model 0 epoch 8 step 3600 loss 3.1300 time 149.5 model 0 done steps 3615 all training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.6 s
- ✓GET /vocab · {"tokens": 32768, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✗POST /score, characters outside the corpus · the response was rejected: VerificationFailure: /score tokens do not concatenate to the exact UTF-8 text. No /vocab token holds the bytes cc ('̃'), cd ('ͱ'), cf ('ς'), d0 ('Р'), d1 ('у'), d2 ('қ'), d3 ('ӣ'), d4 ('Ա'), d5 ('Հ'), d6 ('ր'), d8 ('ا'), da ('ک'), db ('ی'), dc ('ܣ'), dd ('ݢ'), de ('ދ') and 15 more, so no tokenization reproduces text containing them. The token bytes must concatenate to any UTF-8 input, and held-out documents are not limited to the characters of the public data.; got {"token_ids": [181, 152, 150, 150, 119, 117, 118, 26, 147, 95, 159, 114, 157, 161, 147, 115, 114, 147, 114, 158, 166, 147, 162, 151, 117, 154, 147, 121, 164, 112, 119, 117, 154, 147, 153, 149, 109, 122, 155, 152, 116, 150, 119, 117, 154, 147, 110, 152, 120, 123, 119, 12, 147, 113, 109 ...[clipped]
- 3failed9h 0m into the runtook 4m 16s
startup 2.8 s of 300 s; 6/8 gates passed; failed /score vs /dist: max |score - dist| over the same positions; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 202 s, writing 4 files (119768192 bytes) to OUTPUT_DIR. End of its log: n.txt', '/corpus/switchboard.train.txt'] tokenizer vocab 32768 bos_id 0 === model 0 === model 0 epoch 1 step 200 loss 4.6087 time 21.7 model 0 epoch 1 step 400 loss 4.4585 time 29.1 model 0 epoch 2 step 600 loss 3.7797 time 36.5 model 0 epoch 2 step 800 loss 4.2523 time 43.9 model 0 epoch 3 step 1000 loss 3.6993 time 51.3 model 0 epoch 3 step 1200 loss 3.4363 time 58.7 model 0 epoch 3 step 1400 loss 3.8780 time 66.2 model 0 epoch 4 step 1600 loss 3.4229 time 73.6 model 0 epoch 4 step 1800 loss 3.5600 time 81.1 model 0 epoch 5 step 2000 loss 3.4337 time 88.5 model 0 epoch 5 step 2200 loss 3.1384 time 96.0 model 0 epoch 5 step 2400 loss 3.1575 time 103.4 model 0 epoch 6 step 2600 loss 3.2883 time 110.8 model 0 epoch 6 step 2800 loss 3.1261 time 118.2 model 0 epoch 7 step 3000 loss 2.9403 time 125.7 model 0 epoch 7 step 3200 loss 2.9816 time 133.1 model 0 epoch 8 step 3400 loss 2.9801 time 140.5 model 0 epoch 8 step 3600 loss 3.4079 time 148.0 model 0 done steps 3656 all training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.8 s
- ✓GET /vocab · {"tokens": 32768, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": -5.088062327018861e-08, "score_minus_dist": 0.011693120002746582}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.6 s and 86 of 516 /dist and truncation requests in 2.1 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (5 s) as measured
- ✓slowest request · 183 answered requests, median 0.22 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✗/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✗truncated text · max difference over 1093 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed9h 10m into the runtook 4m 19s
startup 3.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 201 s, writing 4 files (119768192 bytes) to OUTPUT_DIR. End of its log: n.txt', '/corpus/switchboard.train.txt'] tokenizer vocab 32768 bos_id 0 === model 0 === model 0 epoch 1 step 200 loss 4.6069 time 20.9 model 0 epoch 1 step 400 loss 4.4588 time 28.3 model 0 epoch 2 step 600 loss 3.7794 time 35.6 model 0 epoch 2 step 800 loss 4.2530 time 43.0 model 0 epoch 3 step 1000 loss 3.6996 time 50.4 model 0 epoch 3 step 1200 loss 3.4355 time 57.8 model 0 epoch 3 step 1400 loss 3.8776 time 65.1 model 0 epoch 4 step 1600 loss 3.4233 time 72.5 model 0 epoch 4 step 1800 loss 3.5597 time 79.9 model 0 epoch 5 step 2000 loss 3.4326 time 87.3 model 0 epoch 5 step 2200 loss 3.1371 time 94.7 model 0 epoch 5 step 2400 loss 3.1569 time 102.1 model 0 epoch 6 step 2600 loss 3.2880 time 109.5 model 0 epoch 6 step 2800 loss 3.1244 time 116.9 model 0 epoch 7 step 3000 loss 2.9390 time 124.3 model 0 epoch 7 step 3200 loss 2.9775 time 131.7 model 0 epoch 8 step 3400 loss 2.9782 time 139.1 model 0 epoch 8 step 3600 loss 3.4070 time 146.5 model 0 done steps 3696 all training done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.0 s
- ✓GET /vocab · {"tokens": 32768, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1227}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 932}
- ✓POST /dist · {"prefix_tokens": 613, "logprobs": 32768, "logsumexp": 8.122369221652548e-08, "score_minus_dist": -1.430511474609375e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.7 s and 86 of 516 /dist and truncation requests in 3.5 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (6 s) as measured
- ✓slowest request · 183 answered requests, median 0.29 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 237 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submission complete.