Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $37.66 of $200
- Model cost
- $16.75
- Agent time
- 4h 52m
- Steps
- 318
- Tokens in / out
- 75.2M / 458k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 12th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4847
- reward
- 0
- nll_nats
- 5314117.7791
- bytes
- 5163651
- documents
- 551
- tokens
- 1325854
- unscored_documents
- 0
- vocab_size
- 16000
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py build
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py builds the GPT and initializes it with nn.init.normal_ (std 0.02, smaller depth-scaled std on output projections). train_llm.py builds the model with GPT(cfg) and never loads a state dict before training. tok_utils.train_tokenizer trains a new byte-level BPE (tokenizers BpeTrainer) on the corpus. No from_pretrained calls or checkpoint loads appear in training. The session shows no model downloads, and only tokenizers, safetensors and s5cmd were installed.
✓no embedded external dataThe shipped files contain no vocabularies, weights, word lists or text tables. The only constants are hyperparameter pre
The shipped files contain no vocabularies, weights, word lists or text tables. The only constants are hyperparameter presets, the standard GPT-2 bytes-to-unicode mapping (an algorithm constant) and a coverage line generated from codepoint ranges so every UTF-8 byte is representable. None of these come from data outside the corpus.
✓training uses only corpustrain.sh copies the submitted sources into $OUTPUT_DIR and runs train_llm.py. That script reads only *.txt files under $
train.sh copies the submitted sources into $OUTPUT_DIR and runs train_llm.py. That script reads only *.txt files under $CORPUS_DIR (tokenizer training, dataset building, lossless-check samples). It writes training.log, dev_docs.jsonl, tokenizer.json, model.safetensors, config.json and architecture.json, all to $OUTPUT_DIR. There is no network use, no reads from other paths and no pretrained loading. The dev split is carved from the corpus itself.
✓serving uses only replay outputsserve.sh runs serve_llm.py with WEIGHTS_DIR. LM.__init__ loads only config.json, tokenizer.json and model.safetensors fr
serve.sh runs serve_llm.py with WEIGHTS_DIR. LM.__init__ loads only config.json, tokenizer.json and model.safetensors from $WEIGHTS_DIR and imports the submitted model.py and tok_utils.py. It does not read the corpus, caches or the network; it only runs a local HTTP server.
✓causal normalized scoringBoth /score and /dist use the same loaded GPT. sliding_logprobs and next_token_logprobs use a causal sliding mask (k<=q,
Both /score and /dist use the same loaded GPT. sliding_logprobs and next_token_logprobs use a causal sliding mask (k<=q, within the last S keys), and padded keys are masked out. /dist returns log_softmax over the full vocabulary (V logits from the tied embedding). Each request is computed fresh under a lock, with no cache of past requests, no lookup of previously seen text and no special-casing of audit requests.
✓no slowrun lookupThe only mentions of Slowrun in the session are the task prompt and the participant reading /task/CONTRACT.md (trajector
The only mentions of Slowrun in the session are the task prompt and the participant reading /task/CONTRACT.md (trajectory#296). The 5 search calls are local regex searches over submission files. There is no web search, git clone or fetch of the slowrun repo or pages about it. The only curl calls download the s5cmd binary from GitHub or hit the local server.
✓harness credentials used only by the harnessNo session step references ANTHROPIC, OPENAI or OPENROUTER keys or base URLs, and no extra harness CLI is launched. The
No session step references ANTHROPIC, OPENAI or OPENROUTER keys or base URLs, and no extra harness CLI is launched. The credentials that do appear are Spin storage session credentials (/tmp/session.json, used with s5cmd, trajectory#129-130) and a local reading of validate.py's judge-key option name (trajectory#301), which is not a use of harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 43m into the runtook 3m 24s
startup 4.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 98 s, writing 13 files (698576899 bytes) to OUTPUT_DIR. End of its log: call last): File "/submission/train_llm.py", line 450, in main save_artifacts(output_dir, base_module, cfg, tok, { File "/submission/train_llm.py", line 528, in save_artifacts with open(os.path.join(output_dir, "architecture.json"), "w", encoding="utf-8") as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/replay/output/architecture.json' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/submission/train_llm.py", line 469, in main save_artifacts(output_dir, base, cfg, tok, {"crashed": True}) File "/submission/train_llm.py", line 528, in save_artifacts with open(os.path.join(output_dir, "architecture.json"), "w", encoding="utf-8") as f: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/replay/output/architecture.json' [10:43:24] done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.8 s
- ✓GET /vocab · {"tokens": 16000, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 959}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16000, "logsumexp": 8.791526662221827e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 21.5 s and 86 of 516 /dist and truncation requests in 4.7 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (11 s) as measured
- ✓slowest request · 183 answered requests, median 0.93 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 486 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed4h 48m into the runtook 3m 35s
startup 7.0 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 111 s, writing 13 files (698576354 bytes) to OUTPUT_DIR. End of its log: rain.txt: lines=531305 groups=2657 dev=26 [10:47:14] gutenberg.train.txt: lines=59934 groups=300 dev=3 [10:47:14] open_subtitles.train.txt: lines=386046 groups=1931 dev=19 [10:47:14] simple_wiki.train.txt: lines=58709 groups=294 dev=2 [10:47:14] switchboard.train.txt: lines=2892 groups=15 dev=0 [10:47:18] model: 174.2M params [10:47:20] optimizer: AdamW {'fused': True} [10:47:20] torch.compile disabled [10:47:20] chunks/epoch~7914 accum=8 eff_batch_tok=262144 [10:47:31] calib step 5/14 loss=7.752 [10:47:42] calib step 10/14 loss=6.951 [10:47:50] calib: 14 steps in 30.1s (2.09s/step steady) -> total_steps=42 warmup=5 eval_every=10 [10:48:03] dev [init] step=14 bpb=2.7355 best=2.7355@14 (IMPROVED; 12.5s) [10:48:03] dev cost 12.5s x~6 -> reserve 94s; total_steps=14 [10:48:15] dev [final] step=14 bpb=2.7355 best=2.7355@14 (no; 12.2s) [10:48:15] trained: steps=14 toks=3.7M epochs=0.2 [10:48:15] restored best dev bpb=2.7355 @ step 14 [10:48:16] saved artifacts (0.8s) [10:48:16] done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 7.0 s
- ✓GET /vocab · {"tokens": 16000, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 959}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16000, "logsumexp": 1.0776123104960789e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 21.6 s and 86 of 516 /dist and truncation requests in 4.8 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (13 s) as measured
- ✓slowest request · 183 answered requests, median 0.70 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1445 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. From-scratch 174M LLaMA-style causal LM (byte-level BPE-16k, ctx 2048, dropout 0.12), selected on a 1% held-out dev split of the training corpus only.
Verified before submit: full 60-min H100 train.sh run; public validation bpb 1.54 with zero unscored docs; two frontier selfcheck passes with all gates green — /score~/dist/truncation audits at exactly 0 nats (bitwise), startup ~5s of 300s, projected eval ~174s of 3600s.