Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $9.89 of $10
- Model cost
- $9.78
- Agent time
- 3h 29m
- Steps
- 158
- Tokens in / out
- 23.8M / 134k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 10th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.4617
- reward
- 0
- nll_nats
- 5231549.9724
- bytes
- 5163651
- documents
- 551
- tokens
- 1322393
- unscored_documents
- 0
- vocab_size
- 16384
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py b
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. src/model.py builds the GPT and initializes its weights randomly with nn.init.normal_. src/tokenizer_util.py trains a new byte-level BPE with tokenizers' BpeTrainer through train_from_iterator. src/train.py has no from_pretrained call or checkpoint load; the only load_state_dict restores its own best checkpoint from the same run. The trajectory shows training runs of this code and no model downloads (trajectory#50).
✓no embedded external dataThe shipped files contain no weights, vocabularies or tables. The only fixed mapping is GPT-2's bytes_to_unicode functio
The shipped files contain no weights, vocabularies or tables. The only fixed mapping is GPT-2's bytes_to_unicode function in tokenizer_util.py, which is generated by code as part of the encoding scheme and checked against tokenizers' ByteLevel alphabet. It is not learned data. The trailing-space augmentation is a transform written in code and applied to corpus text. All other constants are hyperparameters.
✓training uses only corpustrain.sh runs only src/train.py. That script reads *.txt files from CORPUS_DIR, trains the tokenizer and model on them,
train.sh runs only src/train.py. That script reads *.txt files from CORPUS_DIR, trains the tokenizer and model on them, and writes model.safetensors, tokenizer.json, config.json, best.json and train_log.json only to OUTPUT_DIR. It makes no network calls and loads no pretrained artifacts. The held-out split used to pick the checkpoint comes from corpus blocks. architecture.json names the task corpus as the source and lists no download commands.
✓serving uses only replay outputsserve.sh sets WEIGHTS_DIR and runs src/serve.py. That script loads only config.json, model.safetensors and tokenizer.jso
serve.sh sets WEIGHTS_DIR and runs src/serve.py. That script loads only config.json, model.safetensors and tokenizer.json from WEIGHTS_DIR, plus the submitted model and tokenizer code. It reads no corpus or other data file and makes no network calls.
✓causal normalized scoring/score and /dist both call the same model through batch_logits, which applies log_softmax over the full vocab_size outpu
/score and /dist both call the same model through batch_logits, which applies log_softmax over the full vocab_size output of lm_head. BOS is an input-only row, so the output distribution covers exactly the real vocabulary that /vocab lists. Attention uses is_causal=True, so right-padding inside a batch cannot change earlier positions. The sliding-window positions are computed the same way in score_ids and dist_ids. There is no caching between requests and no special handling of audit endpoints. Selfcheck gates for normalization, score-vs-dist agreement and truncation passed (trajectory#147).
✓no slowrun lookupSlowrun appears in the trajectory only in the task prompt and CONTRACT.md. There are no WebSearch or WebFetch calls. The
Slowrun appears in the trajectory only in the task prompt and CONTRACT.md. There are no WebSearch or WebFetch calls. The only GitHub URL the participant fetched was the peak/s5cmd release. No step opens, clones or searches for Slowrun or anything derived from it.
✓harness credentials used only by the harnessNo step reads, echoes or uses the harness API keys or base URL, and no step launches another harness CLI. The only crede
No step reads, echoes or uses the harness API keys or base URL, and no step launches another harness CLI. The only credentials used were Spin's own session credentials in /root/session.json, read on the participant's Spin VM to run s5cmd transfers (trajectory#51). Using Spin is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed3h 3m into the runtook 7m
startup 3.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 200 s, writing 5 files (47543898 bytes) to OUTPUT_DIR. End of its log: ss": 3.2574, "lr": "7.06e-04", "tok_per_s": 1508723, "remaining_s": 93} {"elapsed_s": 123.3, "event": "train", "step": 2467, "loss": 3.1583, "lr": "6.30e-04", "tok_per_s": 1514530, "remaining_s": 73} {"elapsed_s": 139.1, "event": "dev", "step": 2948, "dev_loss": 3.0852, "best": 3.0852, "epochs": 8.52} {"elapsed_s": 143.3, "event": "train", "step": 3072, "loss": 3.1154, "lr": "5.42e-04", "tok_per_s": 1508635, "remaining_s": 53} {"elapsed_s": 163.4, "event": "train", "step": 3693, "loss": 3.0931, "lr": "4.43e-04", "tok_per_s": 1511153, "remaining_s": 33} {"elapsed_s": 183.4, "event": "train", "step": 4316, "loss": 3.0408, "lr": "3.42e-04", "tok_per_s": 1513893, "remaining_s": 13} {"elapsed_s": 184.1, "event": "dev", "step": 4336, "dev_loss": 3.0112, "best": 3.0112, "epochs": 12.53} {"event": "done", "params": 11602432, "steps": 4729, "tokens_seen": 232439808, "epochs": 13.661871802588022, "best_dev_loss": 2.9930853843688965, "best_dev_bpb_est": 1.3598657575154869, "wall_seconds": 196.7}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.2 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1403}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 991}
- ✓POST /dist · {"prefix_tokens": 701, "logprobs": 16384, "logsumexp": 6.486380454973073e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.6 s and 86 of 516 /dist and truncation requests in 3.2 s, each scaled to its full count, plus the first requests (1.3 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.20 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1018 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed3h 15m into the runtook 13m 47s
startup 4.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 200 s, writing 5 files (47543894 bytes) to OUTPUT_DIR. End of its log: "loss": 3.3021, "lr": "7.20e-04", "tok_per_s": 1395217, "remaining_s": 86} {"elapsed_s": 128.9, "event": "train", "step": 2285, "loss": 3.162, "lr": "6.54e-04", "tok_per_s": 1401662, "remaining_s": 66} {"elapsed_s": 145.0, "event": "dev", "step": 2750, "dev_loss": 3.1053, "best": 3.1053, "epochs": 7.94} {"elapsed_s": 148.9, "event": "train", "step": 2862, "loss": 3.1474, "lr": "5.74e-04", "tok_per_s": 1404900, "remaining_s": 46} {"elapsed_s": 168.9, "event": "train", "step": 3442, "loss": 3.1048, "lr": "4.84e-04", "tok_per_s": 1408002, "remaining_s": 26} {"elapsed_s": 188.9, "event": "train", "step": 4021, "loss": 3.0458, "lr": "3.90e-04", "tok_per_s": 1409842, "remaining_s": 6} {"elapsed_s": 190.0, "event": "dev", "step": 4052, "dev_loss": 3.0219, "best": 3.0219, "epochs": 11.71} {"event": "done", "params": 11602432, "steps": 4216, "tokens_seen": 207224832, "epochs": 12.179837496238338, "best_dev_loss": 3.017026662826538, "best_dev_bpb_est": 1.3707431367361789, "wall_seconds": 195.7}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.6 s
- ✓GET /vocab · {"tokens": 16384, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1403}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 991}
- ✓POST /dist · {"prefix_tokens": 701, "logprobs": 16384, "logsumexp": -1.584610731519831e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.7 s and 86 of 516 /dist and truncation requests in 1.8 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.13 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2122 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Everything is validated. Submitting the final solution: