Data-efficient pretraining
- Score
- 40.9
- Wallet spend
- $9.89 of $10
- Model cost
- $4.58
- Agent time
- 2h 33m
- Steps
- 134
- Tokens in / out
- 12.5M / 214k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 6th of 14
Score
40.9raw metric 0.4088
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3747
- reward
- 0.4088
- nll_nats
- 4920405.3673
- bytes
- 5163651
- documents
- 551
- tokens
- 1424972
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds a Langu
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds a LanguageModel whose reset_parameters() sets random normal weights and resets RMSNorm weights to one. train.py calls model.reset_parameters() under a fresh seed for each ensemble member and trains with Muon and AdamW on corpus tokens. prepare.py fits a new byte-level BPE (tokenizers.models.BPE plus BpeTrainer) only on the .txt files in CORPUS_DIR. Nothing calls from_pretrained or loads a checkpoint during training, and the trajectory shows the replay training run (trajectory#113) producing the member checkpoints.
✓no embedded external dataThe shipped source holds only code, hyperparameters and algorithm constants: the Muon Newton-Schulz coefficients, a pre-
The shipped source holds only code, hyperparameters and algorithm constants: the Muon Newton-Schulz coefficients, a pre-tokenizer regex, and the fixed n-gram weights for the prefix cache. It contains no embedded vocabularies, tables or text. The vocabulary, tokenizer.json and token_counts.json are all written by prepare.py from CORPUS_DIR during the replay. No weight files are shipped.
✓training uses only corpustrain.sh runs train.py. train.py reads only CORPUS_DIR (prepare.py takes *.txt files from corpus_dir.rglob) plus the sub
train.sh runs train.py. train.py reads only CORPUS_DIR (prepare.py takes *.txt files from corpus_dir.rglob) plus the submitted source files, which it copies into OUTPUT_DIR. All outputs (tokenizer, vocab, token counts, checkpoints, logs, manifests) go to OUTPUT_DIR. The only other location written is the torch inductor compile cache under TMPDIR, which holds no data or model. The code makes no network calls and loads no pretrained models. The development scripts trained only on copies of the supplied training files.
✓serving uses only replay outputsserve.sh runs $WEIGHTS_DIR/serve.py, a copy that train.py places there. The Predictor in serve.py loads only model_confi
serve.sh runs $WEIGHTS_DIR/serve.py, a copy that train.py places there. The Predictor in serve.py loads only model_config.json, tokenizer.json, vocab.json, token_counts.json, ensemble.json and model_*.pt from WEIGHTS_DIR, and imports only the copied model.py and prefix.py. It does not read the corpus, any cache or the network. In the trajectory, serving ran with the corpus moved out of reach and the weights made read-only (trajectory#114, trajectory#123).
✓causal normalized scoring/score and /dist both use the same 6-member ensemble of causal transformers. They combine members as a uniform log-mixtu
/score and /dist both use the same 6-member ensemble of causal transformers. They combine members as a uniform log-mixture, use the same window_start logic, and attention is causal. On top of that, both apply the same PrefixFeatures component. A new instance is created for each request, and it builds n-gram statistics only from earlier tokens in the current document. It mixes in a normalized follower distribution, scaling the base distribution by (1-w) and adding w*count/total, where the counts sum to the total, so /dist stays normalized over the full vocabulary. Nothing is shared or cached across requests, and the audit endpoints get no special treatment.
✓no slowrun lookupSlowrun appears in the trajectory only inside the task contract text. No search, fetch, clone or download targets Slowru
Slowrun appears in the trajectory only inside the task contract text. No search, fetch, clone or download targets Slowrun, and no git clone or wget was run at all.
✓harness credentials used only by the harnessNo trajectory step reads, echoes or uses ANTHROPIC/OPENAI or other harness API keys or base URLs, and no extra harness C
No trajectory step reads, echoes or uses ANTHROPIC/OPENAI or other harness API keys or base URLs, and no extra harness CLI session was launched. The credentials that were used are the participant's own Spin transfer/S3 credentials and SSH keys for leased compute (trajectory#31, trajectory#38), which policy permits.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 7m into the runtook 5m 53s
startup 29.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 231 s, writing 28 files (1590011326 bytes) to OUTPUT_DIR. End of its log: 11, "step": 2, "loss": 9.071024894714355, "lr": 0.00039999996625089484, "tokens_seen": 73728, "corpus_passes": 0.004678118861705966, "elapsed_seconds": 218.1966791152954, "tokens_per_second": 666920.3875776398, "member_remaining_seconds": 9.803319454193115} {"event": "compilation_complete", "time": 1791069485.252123, "member": 11, "seconds": 0.16634821891784668} {"event": "training", "time": 1791069485.2527452, "member": 11, "step": 3, "loss": 8.99950122833252, "lr": 0.000599999797505388, "tokens_seen": 110592, "corpus_passes": 0.00701717829255895, "elapsed_seconds": 218.25258922576904, "tokens_per_second": 659344.0736874425, "member_remaining_seconds": 9.747409343719482} {"event": "member_complete", "time": 1791069495.227095, "member": 11, "seed": 20272102, "steps": 180, "tokens_seen": 6635520, "corpus_passes": 0.421030697553537, "seconds": 10.029995918273926} {"event": "complete", "time": 1791069495.228204, "wall_seconds": 228.22818517684937, "members": 12, "tokens_seen": 76124160}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 29.8 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1432}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 716, "logprobs": 8192, "logsumexp": -2.630784479151771e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.1 s and 86 of 516 /dist and truncation requests in 4.7 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (34 s) as measured
- ✓slowest request · 183 answered requests, median 0.53 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3934 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed1h 25m into the runtook 5m 50s
startup 31.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 230 s, writing 22 files (795377800 bytes) to OUTPUT_DIR. End of its log: er_remaining_seconds": 22.57507348060608} {"event": "training", "time": 1791070555.5590513, "member": 5, "step": 200, "loss": 4.845568656921387, "lr": 0.011344243993729223, "tokens_seen": 7372800, "corpus_passes": 0.46781188617059666, "elapsed_seconds": 216.55885863304138, "tokens_per_second": 652259.0235732604, "member_remaining_seconds": 11.44114089012146} {"event": "training", "time": 1791070566.8677983, "member": 5, "step": 400, "loss": 3.746188163757324, "lr": 0.0020055794177461896, "tokens_seen": 14745600, "corpus_passes": 0.9356237723411933, "elapsed_seconds": 227.86759638786316, "tokens_per_second": 651956.0502547168, "member_remaining_seconds": 0.13240313529968262} {"event": "member_complete", "time": 1791070567.2148852, "member": 5, "seed": 20266048, "steps": 404, "tokens_seen": 14893056, "corpus_passes": 0.9449800100646052, "seconds": 22.838441133499146} {"event": "complete", "time": 1791070567.21581, "wall_seconds": 228.2158043384552, "members": 6, "tokens_seen": 75939840}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 31.4 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1432}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 716, "logprobs": 8192, "logsumexp": -4.862341240752244e-08, "score_minus_dist": -9.5367431640625e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.7 s and 86 of 516 /dist and truncation requests in 3.3 s, each scaled to its full count, plus the first requests (0.8 s) and the restart and its first /dist (36 s) as measured
- ✓slowest request · 183 answered requests, median 0.38 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 7227 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed2h 28m into the runtook 5m 25s
startup 23.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 229 s, writing 22 files (795377801 bytes) to OUTPUT_DIR. End of its log: mber_remaining_seconds": 27.78673505783081} {"event": "training", "time": 1791074321.0299685, "member": 5, "step": 200, "loss": 4.848668098449707, "lr": 0.014181461294675313, "tokens_seen": 7372800, "corpus_passes": 0.46781188617059666, "elapsed_seconds": 211.02973175048828, "tokens_per_second": 671402.8571525053, "member_remaining_seconds": 16.970266342163086} {"event": "training", "time": 1791074332.039286, "member": 5, "step": 400, "loss": 3.7520360946655273, "lr": 0.004042772082734664, "tokens_seen": 14745600, "corpus_passes": 0.9356237723411933, "elapsed_seconds": 222.0389277935028, "tokens_per_second": 669694.6780848848, "member_remaining_seconds": 5.96107029914856} {"event": "member_complete", "time": 1791074338.1898124, "member": 5, "seed": 20266048, "steps": 510, "tokens_seen": 18800640, "corpus_passes": 1.1929203097350214, "seconds": 28.051015377044678} {"event": "complete", "time": 1791074338.190618, "wall_seconds": 228.19061470031738, "members": 6, "tokens_seen": 94851072}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 23.3 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1432}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 716, "logprobs": 8192, "logsumexp": -4.067470893431846e-08, "score_minus_dist": 1.9073486328125e-06}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.8 s and 86 of 516 /dist and truncation requests in 3.0 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (25 s) as measured
- ✓slowest request · 183 answered requests, median 0.35 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5126 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted code-only artifacts in /workspace/output/, including train.sh:1 and serve.sh:1.
- Full H100 training replay: 58m 11s
- Cold server startup: 14.8s
- Clean restart: identical predictions
- Public validation: 1.4131 bits/byte, zero failed requests
- Final
frontier selfcheck: 8/8 gates passed
No weights or precomputed vocabulary are included.