Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $5.40 of $50
- Model cost
- $39.73
- Agent time
- 8h 46m
- Steps
- 230
- Tokens in / out
- 60.1M / 137k
- Self-checks
- 1 passed of 1 used
- Rank at $50 wallet
- 10th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.5500
- reward
- 0
- nll_nats
- 5547711.9986
- bytes
- 5163651
- documents
- 551
- tokens
- 1286603
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. src/train.py build
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches that. src/train.py builds GPT(cfg) from src/model.py, which initializes Linear and Embedding weights with normal(0, 0.02). The tokenizer is a new tokenizers.BPE trained by BpeTrainer on the corpus files. The code has no from_pretrained call and loads no checkpoint during training. The only torch.load is in serve.py, which reads the model.pt that train.py wrote. Nothing in the trajectory shows a pretrained model being downloaded or used. The trajectory shows training runs of src/train.py from scratch (e.g. trajectory#180) and a final self-check that ran train.sh end to end (trajectory#226).
✓no embedded external dataNone of the shipped source files contains vocabularies, merge lists, weights, word tables or text passages. The only con
None of the shipped source files contains vocabularies, merge lists, weights, word tables or text passages. The only constant-like logic is the GPT-2 byte-to-unicode mapping in _byte_encoder, which is a fixed algorithmic rule for reversible byte encoding, not data learned from anywhere. The vocabulary (vocab.json) and the weights (model.pt) are produced at run time from CORPUS_DIR. The rest is hyperparameters.
✓training uses only corpustrain.sh runs src/train.py. load_corpus reads only the *.txt files under CORPUS_DIR. The tokenizer's temporary copies go
train.sh runs src/train.py. load_corpus reads only the *.txt files under CORPUS_DIR. The tokenizer's temporary copies go to OUTPUT_DIR/_tok_tmp and are deleted afterwards. vocab.json, model.pt, config.json, train_log.jsonl and train.log are all written to OUTPUT_DIR. The training code makes no network calls and loads no pretrained tokenizer or model; the tokenizers library is used only as a BPE trainer. The checkpoint-selection holdout is a set of line blocks taken from the training files themselves. The source of the corpus is declared in data_manifest.json.
✓serving uses only replay outputsserve.sh runs src/serve.py. At startup, Scorer loads only WEIGHTS_DIR/vocab.json and WEIGHTS_DIR/model.pt, both written
serve.sh runs src/serve.py. At startup, Scorer loads only WEIGHTS_DIR/vocab.json and WEIGHTS_DIR/model.pt, both written by train.py, and imports the model code from src/model.py. It does not read the corpus, other data files, caches or the network.
✓causal normalized scoring/score and /dist use the same GPT through Scorer._forward. Text is split into non-overlapping context chunks, each start
/score and /dist use the same GPT through Scorer._forward. Text is split into non-overlapping context chunks, each starting with BOS, and the model uses causal attention (is_causal=True), so each log-prob depends only on earlier tokens in its chunk. /dist rebuilds the same chunk window and returns log_softmax over the full vocabulary. There is no request cache, no lookup of previously seen text, and no special handling of audit requests; each request is computed independently under a lock. The only post-processing is clamping tiny positive values to 0 in /score, which is a numerical guard. In the self-check (trajectory#226), logsumexp normalization, /score vs /dist agreement and the truncation test all measured 0.0 difference.
✓harness credentials used only by the harnessSearches of the trajectory found no API_KEY or BASE_URL references, no model-API calls from the participant's code, no p
Searches of the trajectory found no API_KEY or BASE_URL references, no model-API calls from the participant's code, no printenv, and no extra claude/codex-style harness sessions. The credentials that do appear are Spin wallet storage credentials, used to move the corpus and an rclone binary between the participant's own Spin machines (trajectory#176-179). These are the participant's own Spin allowance, not harness credentials. web_search was the harness's own tool (trajectory#15-29 chunk).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed8h 42m into the runtook 3m 56s
startup 3.2 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 195 s, writing 5 files (55466343 bytes) to OUTPUT_DIR. End of its log: 1130 [12:28:07] step 1200/1509 loss=4.7810 bpb~1.623 lr=1.75e-04 tok/s=681072 [12:28:07] holdout loss=4.1733 bpb~1.416 [12:28:07] new best holdout at step 1200 [12:28:11] step 1250/1509 loss=4.7628 bpb~1.617 lr=1.48e-04 tok/s=635695 [12:28:14] step 1300/1509 loss=4.7556 bpb~1.614 lr=1.25e-04 tok/s=682677 [12:28:18] step 1350/1509 loss=4.7456 bpb~1.611 lr=1.06e-04 tok/s=681520 [12:28:22] step 1400/1509 loss=4.7213 bpb~1.602 lr=9.25e-05 tok/s=680069 [12:28:25] step 1450/1509 loss=4.7276 bpb~1.605 lr=8.37e-05 tok/s=681589 [12:28:29] step 1500/1509 loss=4.7161 bpb~1.601 lr=8.01e-05 tok/s=679160 [12:28:29] step 1509/1509 loss=4.7048 bpb~1.597 lr=8.00e-05 tok/s=678766 [12:28:29] training done steps=1509 elapsed=110.8s best_dev=4.1733 [12:28:29] saving holdout-selected checkpoint [12:28:30] saved /replay/output/model.pt (55.3 MB) [12:28:30] {"event": "done", "steps": 1509, "best_loss": 4.704784207873875, "elapsed": 110.86014866828918, "params": 23468544, "vocab_size": 8192} train.sh done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.2 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1360}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 955}
- ✓POST /dist · {"prefix_tokens": 680, "logprobs": 8192, "logsumexp": -1.1641733888012595e-07, "score_minus_dist": 7.152557373046875e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.0 s and 86 of 516 /dist and truncation requests in 0.9 s, each scaled to its full count, plus the first requests (0.2 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.08 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2341 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The graded package is the seven files in /workspace/output: train.sh, serve.sh, the model and training source, and the data notes.
What it trains is a byte-level BPE tokenizer and a 23M-parameter causal transformer, both from scratch on the BabyLM training files only. Context is 512 tokens. A random 2% of 32-line blocks from those training files is held out only to pick the checkpoint.
The official self-check passed on an H100. Training in the 240-second probe ran at about 680k tokens per second and saved a checkpoint. The server answered /health in 3.2 seconds, and score, distribution, and truncation checks all agreed.
On a 4090, the same recipe scored 1.495 bits per byte on the public validation documents after about 27 minutes. The H100 is roughly twice as fast, so the graded hour should see at least that much training. That is still above the 1.461 reward floor. Longer runs of this size stopped improving, and larger models memorized the corpus.