Data-efficient pretraining
- Score
- 49.5
- Wallet spend
- $40.47 of $200
- Model cost
- $13.78
- Agent time
- 7h 37m
- Steps
- 196
- Tokens in / out
- 19.4M / 283k
- Self-checks
- 1 passed of 1 used
- Rank at $200 wallet
- 6th of 14
Score
49.5raw metric 0.4951
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3565
- reward
- 0.4951
- nll_nats
- 4855296.8318
- bytes
- 5163651
- documents
- 551
- tokens
- 1322393
- unscored_documents
- 0
- vocab_size
- 16385
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. In src/prepare_dat
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. In src/prepare_data.py the byte-level BPE tokenizer is built empty, as Tokenizer(BPE(...)), and trained with BpeTrainer on the corpus text only. The GPT in src/model.py starts from normal(0, 0.02) weights, and src/train.py calls reinit() for each ensemble member. Nothing calls from_pretrained and no checkpoint is loaded during training. The trajectory shows no download of a model.
✓no embedded external dataThe shipped files are source code, shell scripts, logs, manifests and train_stats.json. They contain no vocabulary, weig
The shipped files are source code, shell scripts, logs, manifests and train_stats.json. They contain no vocabulary, weights or tables. The only constants are hyperparameters and the standard GPT-2 byte-to-unicode mapping, which is an algorithmic constant. tokenizer.json, vocab_hex.json and the safetensors files are all written at train time from data under $CORPUS_DIR.
✓training uses only corpustrain.sh runs prepare_data.py, which reads only *.txt files under $CORPUS_DIR, and then train.py, which reads the token
train.sh runs prepare_data.py, which reads only *.txt files under $CORPUS_DIR, and then train.py, which reads the token stream that prepare_data.py packed into $TMPDIR scratch. Serve-time artifacts go to $OUTPUT_DIR. The code makes no network calls and loads no pretrained model or tokenizer. The corpus source is declared in architecture.json and data_manifest.md as the task-provided /task/public/corpus, and the trajectory confirms that location (trajectory#2).
✓serving uses only replay outputsserve.sh runs src/serve.py with --weights-dir $WEIGHTS_DIR. The Engine loads only files from that directory, all written
serve.sh runs src/serve.py with --weights-dir $WEIGHTS_DIR. The Engine loads only files from that directory, all written by train.sh: meta.json, config.json, ensemble.json, vocab_hex.json, tokenizer.json and the model_*.safetensors listed in ensemble.json. It does not read the corpus, any cache or the network.
✓causal normalized scoring/score and /dist both go through score_lib.build_window and ensemble_logprobs_at, with the same window plan and the same
/score and /dist both go through score_lib.build_window and ensemble_logprobs_at, with the same window plan and the same models. Attention uses scaled_dot_product_attention with is_causal=True. In score_plan and dist_plan, target i sees only seq[a_j:i]; any later tokens or padding fall after it, so causal masking hides them. The output is a log_softmax over the full vocabulary, averaged across members and renormalized with logsumexp. Requests are not cached and no endpoint is special-cased.
✓no slowrun lookupThe trajectory mentions Slowrun only in the task prompt, the contract and the participant's own notes restating the ban
The trajectory mentions Slowrun only in the task prompt, the contract and the participant's own notes restating the ban (trajectory#1, trajectory#2). Every tool call is Bash, Edit, Write or Read; there is no WebSearch or WebFetch. The only external fetches are pypi and the s5cmd release from GitHub. Nothing targets the qlabs-eng/slowrun repository or pages about it.
✓harness credentials used only by the harnessNo step reads, echoes or uses ANTHROPIC/OPENAI-style API keys or harness base URLs, and no extra harness CLI sessions ar
No step reads, echoes or uses ANTHROPIC/OPENAI-style API keys or harness base URLs, and no extra harness CLI sessions are launched. The S3 credentials the participant used came from `spin transfer credentials` and were used on their own Spin leases, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed7h 25m into the runtook 3m 14s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 122 s, writing 9 files (163436820 bytes) to OUTPUT_DIR. End of its log: to 50 [train 73.52s] member 0: 50 steps (0.4 epochs), ema halflife 20, warmup 10 [train 75.51s] member 0: hard deadline reached at step 14/50 [train 76.02s] member 0 done in 2s; chosen model_0_ema.safetensors [train 76.02s] ALL DONE in 76s; members: ['model_0_ema.safetensors'] == train.sh done at 1791552497 (deadline 1791552614, margin 117s) == total 159606 drwx------ 1 nobody nogroup 200 Oct 9 13:28 . drwxr-xr-x 1 root root 120 Oct 9 13:26 .. -rw-r--r-- 1 nobody nogroup 190 Oct 9 13:26 config.json -rw-r--r-- 1 nobody nogroup 234 Oct 9 13:28 ensemble.json drwxr-xr-x 1 nobody nogroup 80 Oct 9 13:26 logs -rw-r--r-- 1 nobody nogroup 577 Oct 9 13:26 meta.json -rw-r--r-- 1 nobody nogroup 162063800 Oct 9 13:28 model_0_ema.safetensors -rw-r--r-- 1 nobody nogroup 1119192 Oct 9 13:26 tokenizer.json -rw-r--r-- 1 nobody nogroup 1006 Oct 9 13:28 train_stats.json -rw-r--r-- 1 nobody nogroup 249568 Oct 9 13:26 vocab_hex.json
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 16385, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16385, "logsumexp": 6.818265552155367e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 10.3 s and 86 of 516 /dist and truncation requests in 2.2 s, each scaled to its full count, plus the first requests (0.9 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.26 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 6541 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Serving path verified clean on re-read. Check selfcheck progress: