Data-efficient pretraining
- Score
- 58.2
- Wallet spend
- $10.47 of $50
- Model cost
- $14.55
- Agent time
- 2h 47m
- Steps
- 414
- Tokens in / out
- — / —
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 6th of 19
Score
58.2raw metric 0.5819
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3382
- reward
- 0.5819
- nll_nats
- 4789740.8523
- bytes
- 5163651
- documents
- 551
- tokens
- 1427990
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py trains a n
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches. train.py trains a new ByteLevel BPE tokenizer by calling models.BPE() and BpeTrainer on $CORPUS_DIR/*.train.txt. It builds the GPT models from model.py, whose _init_weights sets weights from normal(0, 0.02) or zeros. Nothing in the code loads a checkpoint or uses from_pretrained before training. The trajectory shows only pip installs of the tokenizers, safetensors and numpy libraries, with no model downloads.
✓no embedded external dataThe source files contain only code, hyperparameters (MODEL_SPECS, learning rates, seeds, temperature candidates) and alg
The source files contain only code, hyperparameters (MODEL_SPECS, learning rates, seeds, temperature candidates) and algorithm constants such as the Newton-Schulz coefficients and the GPT-2 byte-to-unicode mapping. The vocabulary, merges and weights are all made at runtime from the corpus. No embedded tables, vocabularies, text passages or encoded data were found.
✓training uses only corpustrain.sh runs train.py, which reads only $CORPUS_DIR/*.train.txt (falling back to *.txt in the same directory). It write
train.sh runs train.py, which reads only $CORPUS_DIR/*.train.txt (falling back to *.txt in the same directory). It writes tokenizer.json, vocab_hex.json, model_*.safetensors, ensemble_config.json and training_log.json only to $OUTPUT_DIR, and tee writes train_stdout.log there too. There is no network use and no pretrained loading. Calibration and selection use out-of-fold splits taken from $CORPUS_DIR.
✓serving uses only replay outputsserve.py reads only vocab_hex.json, tokenizer.json, ensemble_config.json and the model safetensors files listed in it, a
serve.py reads only vocab_hex.json, tokenizer.json, ensemble_config.json and the model safetensors files listed in it, all from $WEIGHTS_DIR. It builds the GPT models from the submitted model.py. It does not read the corpus, a cache or the network.
✓causal normalized scoring/score and /dist both use the same ensemble of causal GPT models. Attention uses is_causal=True, and each model's log-pr
/score and /dist both use the same ensemble of causal GPT models. Attention uses is_causal=True, and each model's log-probs come from log_softmax over the full vocab_size logits (BOS is left out of the output). The ensemble combines them with a weighted logsumexp, so the result is a normalized mixture. Both endpoints use the same windowing logic. The optional cache is computed only from the current request's prefix, and train.py sets cache_alpha to 0.0, so it is turned off. There is no request-history state and no special handling of audit endpoints.
✓no slowrun lookupSlowrun appears only in the task prompt and CONTRACT.md (trajectory#1, trajectory#2). The session had no web search or f
Slowrun appears only in the task prompt and CONTRACT.md (trajectory#1, trajectory#2). The session had no web search or fetch tools (only run_command, write_to_file, view_file, replace_file_content and manage_task). The only GitHub access was downloading the s5cmd release on a Spin machine (trajectory#28). Nothing shows the Slowrun repository, its forks or pages about it being accessed.
✓harness credentials used only by the harnessNo step reads, echoes or uses harness API keys or a base URL, and no other harness CLI was launched. The S3 credentials
No step reads, echoes or uses harness API keys or a base URL, and no other harness CLI was launched. The S3 credentials used in trajectory#28 are temporary Spin transfer credentials from the participant's own wallet, used to move the task bundle to the Spin machine, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 29m into the runtook 3m 40s
startup 5.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 159 s, writing 6 files (183218467 bytes) to OUTPUT_DIR. End of its log: Trained ByteLevel BPE tokenizer (V=8192) in 3.82s Encoded 17,238,443 tokens across 1,104,106 lines from 6 files in 8.48s [1/7] Training m0_768_L12_2k (91,275,444 params, seq_len=2048, steps=1300) step 601/1300 | loss 2.4959 | fast_oof_bpb 1.26362 | 133.2s Reached hard deadline inside step loop; finalizing current model. Finished m0_768_L12_2k in 138.3s | best_temp=1.09 | oof_bpb=1.26311 Stopping before m1_896_L14_2k to respect wall-clock budget (remaining: -127s) Completed training 1 models in 150.8s total.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.6 s
- ✓GET /vocab · {"tokens": 8192, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -4.874897752138452e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 8.0 s and 86 of 516 /dist and truncation requests in 1.9 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.22 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 256 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed2h 42m into the runtook 3m 27s
startup 5.1 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 139 s, writing 6 files (183218485 bytes) to OUTPUT_DIR. End of its log: Trained ByteLevel BPE tokenizer (V=8192) in 3.53s Encoded 17,238,443 tokens across 1,104,106 lines from 6 files in 9.62s [1/8] Training m0_768_L12_2k (91,275,444 params, seq_len=2048, steps=1150) step 501/1150 | loss 2.6690 | fast_oof_bpb 1.31431 | 114.3s Reached hard deadline inside step loop; finalizing current model. Finished m0_768_L12_2k in 118.7s | best_temp=1.08 | oof_bpb=1.31215 Stopping before m1_896_L14_2k to respect wall-clock budget (remaining: -107s) Completed training 1 models in 132.0s total.
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 5.1 s
- ✓GET /vocab · {"tokens": 8192, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -1.5589219470868443e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 6.4 s and 86 of 516 /dist and truncation requests in 2.0 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.24 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2001 shared positions of 2 truncated texts
- ✓healthy after requests