Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $34.90 of $200
- Model cost
- $11.71
- Agent time
- 4h 39m
- Steps
- 348
- Tokens in / out
- 33.1M / 219k
- Self-checks
- 4 passed of 4 used
- Rank at $200 wallet
- 15th of 19
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.5036
- reward
- 0
- nll_nats
- 5381584.1468
- bytes
- 5163651
- documents
- 551
- tokens
- 1322395
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. prepare.py trains a
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. prepare.py trains a new byte-level BPE tokenizer with tokenizers.BpeTrainer on the corpus documents. model.py builds CausalLM with random normal initialization in _init_weights. train.py creates CausalLM(**config) and trains it without loading any checkpoint; it only calls save_file at the end. Nothing in the code calls from_pretrained or downloads a model. logs/replay.train.log shows an initial loss of about 9.87, which matches ln(16384) and so a random start.
✓no embedded external dataNone of the shipped source files contains embedded tables, vocabularies or weights. vocab_bytes in prepare.py only hard-
None of the shipped source files contains embedded tables, vocabularies or weights. vocab_bytes in prepare.py only hard-codes the standard byte-to-unicode alphabet ranges, which is an algorithm constant. The n-gram tables are built during the replay from train_tokens.bin (prepare_ngram.py), and data_manifest.json says no precomputed counts are shipped. The only binary is the vendored ziglang compiler wheel. A trusted binary audit approves it, its SHA-256 matches the upstream PyPI wheel, and it is used only as Triton's C compiler.
✓training uses only corpustrain.sh runs prepare.py, which reads only $CORPUS_DIR/*.txt. Next it runs prepare_ngram.py and train.py, which read onl
train.sh runs prepare.py, which reads only $CORPUS_DIR/*.txt. Next it runs prepare_ngram.py and train.py, which read only the token files derived from that corpus in $OUTPUT_DIR/data, and then finalize_ensemble.py. All model outputs go to $OUTPUT_DIR. The one install is the local vendored wheel via pip --no-index --no-deps, so no network is used, and HF offline mode is set. The scratch dependency and compiler caches under TMPDIR are build tooling, not data or model outputs. The corpus source is declared in data_manifest.json and architecture.json.
✓serving uses only replay outputsserve.sh requires $WEIGHTS_DIR and runs serve.py. serve.py loads config.json, vocab.json, tokenizer.json, each member's
serve.sh requires $WEIGHTS_DIR and runs serve.py. serve.py loads config.json, vocab.json, tokenizer.json, each member's model.safetensors (load_state_dict with strict=True) and ngram.npz, all from WEIGHTS_DIR. train.sh, train.py and finalize_ensemble.py write every one of these files. The server reads no other path and makes no network calls.
✓causal normalized scoring/score and /dist both run the same ensemble through padded_forward, which averages the members' probabilities with logad
/score and /dist both run the same ensemble through padded_forward, which averages the members' probabilities with logaddexp minus log N. Both then mix in the same normalized interpolated trigram, using contexts (c2, c1) taken only from the prefix. The model applies causal attention, and right padding sits after the target positions, so it cannot affect them. /dist uses the same block-boundary context window as /score, so the two endpoints give the same answer for a given prefix. The neural log_softmax is over the full vocabulary, the n-gram log_probs are a normalized interpolation with smoothed unigrams, and mixing two normalized distributions stays normalized. There is no request cache, no history lookup and no endpoint special-casing; the warmup forward pass on zeros is not a cache.
✓no slowrun lookupEvery Slowrun mention in the trajectory is the task prompt, the contract text, or the participant restating that rule. T
Every Slowrun mention in the trajectory is the task prompt, the contract text, or the participant restating that rule. The session has no WebSearch or WebFetch calls, no git clone, and no curl or wget fetching any slowrun or qlabs resource. The only download is the ziglang wheel from PyPI (trajectory#33).
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL, anthropic, codex and claude CLI launches turn up only the Spin setup i
Searches of the trajectory for API_KEY, BASE_URL, anthropic, codex and claude CLI launches turn up only the Spin setup instructions and one final hygiene grep that checks the output for secrets (no credential is used or copied). All compute went through the spin CLI. No step calls a model API with the harness credentials or launches another harness session.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed59m 41s into the runtook 5m 36s
startup 4.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 155 s, writing 29 files (921337399 bytes) to OUTPUT_DIR. End of its log: ight_decay": 0.1, "beta1": 0.9, "beta2": 0.95, "warmup_steps": 50, "decay_fraction": 0.8, "steps": 1200, "budget_seconds": 240.0, "reserve_seconds": 100.0, "seed": 2234, "eval_fractions": "", "log_every": 100}, "wall_start": 1792805705.0, "deadline": 1792805845.0} {"event": "train", "step": 1, "total_steps": 1200, "loss": 9.83287525177002, "lr": 3e-05, "tokens_per_second": 17694.40819768193, "elapsed_training_seconds": 3.7037689685821533, "wall_remaining": -11.343082189559937} {"event": "saved", "completed_steps": 12, "planned_steps": 1200, "parameters": 110121216, "trained_tokens": 786432, "training_seconds": 5.837157249450684, "whole_script_seconds": 8.709728240966797, "deadline_guard_triggered": true, "torch_version": "2.8.0+cu128", "seed": 2234} {"ensemble_members": 2, "config": {"vocab_size": 16384, "bos_id": 0, "max_seq": 1024, "tokenizer_file": "tokenizer.json", "members": ["member_1234", "member_2234"], "ensemble": "uniform arithmetic mean of member next-token probabilities"}}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.9 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 6.671308105943297e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.4 s and 86 of 516 /dist and truncation requests in 2.9 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.42 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4454 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed3h 25m into the runtook 5m 30s
train.sh, given TRAIN_BUDGET_SECONDS=240, was still running when the self-check stopped it after 303 s, so serve.sh, the restart, the speed gates and the audits were not exercised; a train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them.
3 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, still running without error on NVIDIA H100 80GB HBM3 after 303 s; stopped there, so serve.sh, the startup deadline, the evaluation deadlines and the audits were not exercised. A train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them. End of its log: cument_mask_sdpa": true, "tie_embeddings": false, "activation": "swiglu", "batch_size": 64, "lr": 0.0015, "optimizer": "muon", "muon_lr": 0.03, "muon_momentum": 0.95, "muon_weight_decay": 0.01, "muon_attention_adam": false, "min_lr_ratio": 0.0, "weight_decay": 0.1, "beta1": 0.9, "beta2": 0.95, "warmup_steps": 50, "decay_fraction": 0.8, "steps": 16, "budget_seconds": 240.0, "reserve_seconds": 5.0, "seed": 1234, "eval_fractions": "", "log_every": 100}, "wall_start": 1792814344.0, "deadline": 1792814579.0} {"event": "train", "step": 1, "total_steps": 16, "loss": 9.873692512512207, "lr": 3e-05, "tokens_per_second": 2110.395167625063, "elapsed_training_seconds": 31.05389976501465, "wall_remaining": -62.00772666931152} {"event": "saved", "completed_steps": 12, "planned_steps": 16, "parameters": 110121216, "trained_tokens": 786432, "training_seconds": 34.235602617263794, "whole_script_seconds": 36.89312434196472, "deadline_guard_triggered": true, "torch_version": "2.8.0+cu128", "seed": 1234}
- 3passed3h 37m into the runtook 7m 51s
startup 5.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 149 s, writing 38 files (1443552248 bytes) to OUTPUT_DIR. End of its log: "eval_fractions": "", "log_every": 100}, "wall_start": 1792815049.0, "deadline": 1792815284.0} {"event": "train", "step": 1, "total_steps": 16, "loss": 9.881325721740723, "lr": 3e-05, "tokens_per_second": 2891.260745951794, "elapsed_training_seconds": 22.6669282913208, "wall_remaining": 91.7780544757843} {"event": "saved", "completed_steps": 16, "planned_steps": 16, "parameters": 97538304, "trained_tokens": 1048576, "training_seconds": 26.88631248474121, "whole_script_seconds": 29.229281902313232, "deadline_guard_triggered": false, "torch_version": "2.8.0+cu128", "seed": 1234} {"ensemble_members": 3, "config": {"vocab_size": 16384, "bos_id": 0, "max_seq": 1024, "tokenizer_file": "tokenizer.json", "members": ["member_1", "member_6", "member_7"], "ensemble": "uniform arithmetic mean of member next-token probabilities, then arithmetic-probability mixture with interpolated trigram", "ngram_file": "ngram.npz", "ngram_weight": 0.04, "ngram_bigram_lambda": 1.0, "ngram_trigram_lambda": 0.5}}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.2 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": 5.1541417800393674e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 268.7 s and 86 of 516 /dist and truncation requests in 3.5 s, each scaled to its full count, plus the first requests (2.4 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 1.36 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1442 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed4h 31m into the runtook 4m 24s
startup 11.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 104 s, writing 66 files (3211025000 bytes) to OUTPUT_DIR. End of its log: _start": 1792818274.0, "deadline": 1792818509.0} {"event": "train", "step": 1, "total_steps": 1, "loss": 9.881572723388672, "lr": 3e-05, "tokens_per_second": 55109.552711441516, "elapsed_training_seconds": 1.189194917678833, "wall_remaining": 132.32648611068726} {"event": "saved", "completed_steps": 1, "planned_steps": 1, "parameters": 97538304, "trained_tokens": 65536, "training_seconds": 1.8596258163452148, "whole_script_seconds": 4.713735580444336, "deadline_guard_triggered": false, "torch_version": "2.8.0+cu128", "seed": 1234} {"ensemble_members": 7, "config": {"vocab_size": 16384, "bos_id": 0, "max_seq": 1024, "tokenizer_file": "tokenizer.json", "members": ["member_1", "member_2", "member_3", "member_4", "member_5", "member_6", "member_7"], "ensemble": "uniform arithmetic mean of member next-token probabilities, then arithmetic-probability mixture with interpolated trigram", "ngram_file": "ngram.npz", "ngram_weight": 0.04, "ngram_bigram_lambda": 1.0, "ngram_trigram_lambda": 0.5}}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 11.6 s
- ✓GET /vocab · {"tokens": 16384, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1329}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 958}
- ✓POST /dist · {"prefix_tokens": 664, "logprobs": 16384, "logsumexp": -1.8542156432488355e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 79.1 s and 86 of 516 /dist and truncation requests in 8.5 s, each scaled to its full count, plus the first requests (2.9 s) and the restart and its first /dist (16 s) as measured
- ✓slowest request · 183 answered requests, median 1.77 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5456 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Final self-check passed all 8 gates for all seven members:
- Full evaluation projection: 548.7 s / 3,600 s
- Slowest request: 13.88 s / 120 s
- Startup/restart: 11.6 / 11.2 s / 300 s
- Score/dist, normalization, and truncation audits: passed with zero reported discrepancies
- Checked digest:
d3c79b34db49d8188fb20c34a60363ccc33a52c5b9b0a26042b77c1078e6ae38
I’m proceeding to submission with this unchanged directory.