Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $8.82 of $10
- Model cost
- $8.85
- Agent time
- 8h 7m
- Steps
- 214
- Tokens in / out
- 45.8M / 256k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 10th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4864
- reward
- 0
- nll_nats
- 5319974.6831
- bytes
- 5163651
- documents
- 551
- tokens
- 1322450
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py b
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py builds a CausalLM in plain PyTorch and initializes it with trunc_normal_ and ones_ in reset_parameters. src/tok_util.py trains a new BPE tokenizer with the tokenizers library's BpeTrainer.train_from_iterator, using only the corpus text. train.py builds the model with build_model(cfg) and never loads outside weights. The only torch.load reads its own logs/best.pt checkpoint, which it wrote earlier in the same run. The trajectory has no from_pretrained calls and no model downloads.
✓no embedded external dataThe shipped files are source code only (8 files, 40.2 KB according to trajectory#212). The only constants are hyperparam
The shipped files are source code only (8 files, 40.2 KB according to trajectory#212). The only constants are hyperparameters, the GPT-2 byte-to-unicode mapping (an algorithm constant), a few short round-trip test strings, and format strings. No vocabularies, word tables, weights or encoded data are embedded.
✓training uses only corpustrain.sh runs src/train.py with --corpus $CORPUS_DIR and --output $OUTPUT_DIR. train.py reads only $CORPUS_DIR/*.txt and
train.sh runs src/train.py with --corpus $CORPUS_DIR and --output $OUTPUT_DIR. train.py reads only $CORPUS_DIR/*.txt and writes the tokenizer, weights, manifests, logs and source copies into the output directory. It sets HF_HUB_OFFLINE and keeps JIT caches under $OUTPUT_DIR. The source copy reads the submission's own scripts, which counts as submitted code. The code makes no network calls. Held-out model selection uses the last 1% of each corpus file, not validation.jsonl.
✓serving uses only replay outputsserve.sh runs src/server.py --weights $WEIGHTS_DIR. The server loads only config.json, tokenizer/tokenizer.json and mode
serve.sh runs src/server.py --weights $WEIGHTS_DIR. The server loads only config.json, tokenizer/tokenizer.json and model.safetensors from that directory. All three are written by train.py. It makes no network, corpus or other data access.
✓causal normalized scoringBoth /score and /dist use the same model through src/scoring.py. score_ids and dist_for_prefix share the ctx_start windo
Both /score and /dist use the same model through src/scoring.py. score_ids and dist_for_prefix share the ctx_start windowing rule and use causal SDPA (is_causal=True), so each log-prob depends only on the prefix. /dist returns a full log_softmax over the whole vocabulary. The server is stateless: it has no request-history cache and no special-casing of the audit endpoints. The lock only serializes GPU access.
✓harness credentials used only by the harnessThe trajectory shows no use of harness API keys or base URLs. The only API_KEY and BASE_URL matches are in spin document
The trajectory shows no use of harness API keys or base URLs. The only API_KEY and BASE_URL matches are in spin documentation output. Remote work used Spin compute and Spin transfer credentials to move the task corpus and code to a leased H100 (trajectory#174, trajectory#175), which is allowed. No extra harness CLI sessions were launched and no credentials were echoed or written to files.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed7h 50m into the runtook 6m 7s
startup 3.5 s of 300 s; 1/1 gates passed; failed POST /dist: the response was rejected: VerificationFailure: /score disagrees with /dist by 0.0304 nats (> 0.02) at token 664; got {"logprobs": [-9.258564949035645, -6.7761430740356445, -6.1667680740356445, -10.065205574035645, -10.627705574035
9 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 302 s, writing 976 files (284102463 bytes) to OUTPUT_DIR. That is 62 s past its budget: if it times its run from TRAIN_BUDGET_SECONDS, the same overrun past 3600 s fails grading. End of its log: train.sh: corpus=/corpus output=/replay/output budget=240s python=/opt/conda/bin/python [step -1] startup corpus=/corpus budget_s=240.0 margin_s=420 device=cuda torch=2.8.0+cu128 cuda=12.8 gpu=NVIDIA H100 80GB HBM3 [step -1] corpus files=6 train_bytes=53856305 heldout_bytes=543535 [step -1] tokenizer vocab=16384 bos=<bos> [step -1] tokenized train_tokens=16162093 bytes_per_token=3.332 [step -1] model params=55591424 compile=default batch_seqs=32 micro_seqs=8 accum=4 [step 8] benchmark compile_s=82.8 step_s=0.322 toks_per_s=407055 [step 8] heldout bpb=2.9408 note=init per_file={'bnc_spoken.train.txt': 2.522, 'childes.train.txt': 3.051, 'gutenberg.train.txt': 2.828, 'open_subtitles.train.txt': 2.955, 'simple_wiki.train.txt': 3.103, 'switchboard.train.txt': 2.991} [step 8] heldout bpb=2.9408 best=2.9408 improved=False note=final [step 8] done_train steps=8 epochs=0.06 best_step=8 elapsed_s=295 [step 8] saved elapsed_s=296
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.5 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1328}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✗POST /dist · the response was rejected: VerificationFailure: /score disagrees with /dist by 0.0304 nats (> 0.02) at token 664; got {"logprobs": [-9.258564949035645, -6.7761430740356445, -6.1667680740356445, -10.065205574035645, -10.627705574035645, -6.9714555740356445, -10.557393074035645, -6.8230180740356445, -9.545674324035645, -7.3698930740356445, -4.8230180740356445, -10.627705574035645, -3.7917683124542236, ...[clipped]
- 2passed8h 1m into the runtook 4m 30s
startup 2.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 178 s, writing 976 files (284100054 bytes) to OUTPUT_DIR. End of its log: train.sh: corpus=/corpus output=/replay/output budget=240s python=/opt/conda/bin/python [step -1] startup corpus=/corpus budget_s=240.0 margin_s=420 device=cuda torch=2.8.0+cu128 cuda=12.8 gpu=NVIDIA H100 80GB HBM3 [step -1] corpus files=6 train_bytes=53856305 heldout_bytes=543535 [step -1] tokenizer vocab=16384 bos=<bos> [step -1] tokenized train_tokens=16162093 bytes_per_token=3.332 [step -1] model params=55591424 compile=default batch_seqs=32 micro_seqs=8 accum=4 [step 8] benchmark compile_s=41.8 step_s=0.32 toks_per_s=409081 [step 8] heldout bpb=2.9408 note=init per_file={'bnc_spoken.train.txt': 2.522, 'childes.train.txt': 3.052, 'gutenberg.train.txt': 2.828, 'open_subtitles.train.txt': 2.955, 'simple_wiki.train.txt': 3.102, 'switchboard.train.txt': 2.991} [step 8] heldout bpb=2.9408 best=2.9408 improved=False note=final [step 8] done_train steps=8 epochs=0.06 best_step=8 elapsed_s=172 [step 8] saved elapsed_s=173
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.8 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1328}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.3857680247753024e-07, "score_minus_dist": 9.5367431640625e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.4 s and 86 of 516 /dist and truncation requests in 15.3 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.64 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2452 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final summary:
Result: from-scratch causal LM (byte-level BPE-16k + 56M decoder-only transformer) trained on the 10M-word corpus only. Rehearsal on H100 reached 1.273 bits/byte on my corpus held-out (reward ≈ 0.90; floor 1.461, ceiling 1.25). Selfcheck passed all gates.
Submission (/workspace/output, 8 files): train.sh (fills any TRAIN_BUDGET_SECONDS via wall-clock cosine schedule + best-checkpoint selection on a 1% corpus tail), serve.sh (stdlib HTTP server), src/ (model, training, scoring, tokenizer, server), architecture.json (base_model="none", base_revision="from-scratch").
Key findings from ablations (all recipe decisions on my own held-out slice, never the task validation set):
- LR 8e-4 + dropout 0.15 won; higher LRs and dropout 0 overfit by ~12–16 epochs; 100M overfit earlier than 56M at every LR.
- Best-selection was essential: 51-epoch runs hump after ~15–20 epochs but keep the early best.
Bug caught by selfcheck #1: /score↔/dist differed by 0.03 nats on CUDA — bf16 SDPA rounding is forward-length-dependent. Fixed by serving in fp32 plus a request lock; verified max diff 1.2e-05 over 302 positions on H100, then selfcheck #2 passed everything (score−dist 9.5e-07, startup 2.8s, slowest request 2.06s).
Notes: no test files ship with the submission (no harness exists there; verified via unit probes, H100 audits, rehearsal, and selfcheck instead). Spend $8.81 of $10.