Data-efficient pretraining
- Score
- 0.0
- Wallet spend
- $7.29 of $10
- Model cost
- $44.94
- Agent time
- 4h 1m
- Steps
- 273
- Tokens in / out
- 61.8M / 158k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 10th of 14
Score
0.0raw metric 0
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.5496
- reward
- 0
- nll_nats
- 5546360.7895
- bytes
- 5163651
- documents
- 551
- tokens
- 5163651
- unscored_documents
- 0
- vocab_size
- 256
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. train.py builds ByteLM(
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. train.py builds ByteLM(cfg), calls model.init_weights() (normal and zero init in model.py) and trains it with AdamW. The tokenizer is the identity map over the 256 byte values, written by save_all. No from_pretrained call, checkpoint load or distillation appears anywhere in the training path. I found no trajectory step that downloads a pretrained model to initialize the shipped model.
✓no embedded external dataThe shipped files are only code, configs and short manifests or logs. They contain no embedded weights, vocabularies, n-
The shipped files are only code, configs and short manifests or logs. They contain no embedded weights, vocabularies, n-gram tables or text passages. The 'n-gram' features are learned hashed embedding tables that start random (nn.init.normal_) and are trained on the corpus. tokenizer.json is generated as hex strings for bytes 0-255. The constants that remain are hyperparameters (the RoPE base and the hash multiplier 257).
✓training uses only corpustrain.sh only runs train.py. load_corpus reads *.txt files from CORPUS_DIR, and every output (config.json, model.pt, tok
train.sh only runs train.py. load_corpus reads *.txt files from CORPUS_DIR, and every output (config.json, model.pt, tokenizer.json, train_summary.json, train_log.jsonl) is written under OUTPUT_DIR. The imports are torch, json, math, os and time, with no network use and no pretrained loaders. The data source (the task corpus at CORPUS_DIR) is declared in data_manifest.json and data_download.log. The trajectory shows validation.jsonl being read only by separate dev benchmarking scripts, never by train.py.
✓serving uses only replay outputsserve.sh runs serve.py. serve.py's load() reads only WEIGHTS_DIR/config.json, WEIGHTS_DIR/model.pt (with weights_only=Tr
serve.sh runs serve.py. serve.py's load() reads only WEIGHTS_DIR/config.json, WEIGHTS_DIR/model.pt (with weights_only=True) and WEIGHTS_DIR/tokenizer.json, plus the submitted model.py. It does not touch the corpus, other paths or the network.
✓causal normalized scoring/score (score_ids) and /dist (next_logprobs) use the same MODEL forward, with a causal SDPA (is_causal=True) and log_sof
/score (score_ids) and /dist (next_logprobs) use the same MODEL forward, with a causal SDPA (is_causal=True) and log_softmax over all 256 byte logits. Both build blocks the same way: the first block starts from sentinel 256, and later blocks start at the byte just before the block boundary. Each log-prob therefore depends only on the prefix. The hashed n-gram embedding uses only the current and previous ids. There is no cache, no request history and no special-casing. An intra-request cache benchmark appears only in a dev script (/tmp/dev/bench_cache.py) and was not shipped.
✓harness credentials used only by the harnessTrajectory searches for API_KEY, BASE_URL, openrouter and claude found nothing. Compute was used through the spin CLI, a
Trajectory searches for API_KEY, BASE_URL, openrouter and claude found nothing. Compute was used through the spin CLI, and the AWS-style keys came from the spin transfer session.json, not the harness credentials. I found no step where harness credentials were used, copied or echoed, or where extra harness CLIs were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed3h 56m into the runtook 3m 28s
startup 3.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
17 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 132 s, writing 5 files (168881974 bytes) to OUTPUT_DIR. End of its log: 0.00199 82s tok/s 904138 step 800 loss 1.086 ema_bpb 1.596 lr 0.00183 85s tok/s 929233 step 850 loss 1.075 ema_bpb 1.570 lr 0.00167 88s tok/s 952621 step 900 loss 1.067 ema_bpb 1.549 lr 0.00151 91s tok/s 974331 step 950 loss 1.047 ema_bpb 1.529 lr 0.00134 94s tok/s 994532 step 1000 loss 1.023 ema_bpb 1.508 lr 0.00118 97s tok/s 1013579 curve bpb 1.5232 on 20480 bytes saved weights to /replay/output step 1000 step 1050 loss 1.045 ema_bpb 1.493 lr 0.00103 100s tok/s 1028437 step 1100 loss 1.032 ema_bpb 1.482 lr 0.00088 103s tok/s 1045296 step 1150 loss 1.035 ema_bpb 1.466 lr 0.00074 106s tok/s 1061078 step 1200 loss 0.997 ema_bpb 1.449 lr 0.00061 110s tok/s 1075990 step 1250 loss 0.983 ema_bpb 1.439 lr 0.00050 113s tok/s 1089995 step 1300 loss 1.002 ema_bpb 1.432 lr 0.00040 116s tok/s 1103290 step 1350 loss 1.012 ema_bpb 1.427 lr 0.00031 119s tok/s 1115900 saved weights to /replay/output step 1368 done steps 1368 tokens 134479872 elapsed 120.0s train.sh done
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.4 s
- ✓GET /vocab · {"tokens": 256, "special": 0}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 5203}
- ✓POST /dist · {"prefix_tokens": 2601, "logprobs": 256, "logsumexp": 2.2451411174628788e-08, "score_minus_dist": 0.0002307891845703125}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 7.1 s and 86 of 516 /dist and truncation requests in 1.8 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 182 answered requests, median 0.37 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 22973 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. The grading check passed before hand-in, and the session is now closed.
The submission is a byte-level causal transformer trained from scratch on the six corpus files only. The tokenizer is the 256 byte values, written by train.sh. A document starts from a learned sentinel that is not part of the text vocabulary.
On the public validation set this setup scored about 1.54 bits per byte. That is still above the 1.461 floor, so the linear reward is zero. A larger model memorized the 54 MB corpus (training loss near 1.0, validation worse), so the submitted run is the smaller one that generalized: about 40M parameters, dropout, and a learning-rate cosine that stops after roughly ten epochs.
frontier selfcheck passed on an H100:
train.shfinished a 240-second probe in 132 seconds and wrote the weights.- The server answered
/healthin 3.4 seconds. /scoreand/distagreed within 0.0007 nats, and a truncated text kept its shared log-probabilities within 0.0005 nats.- The slowest request was 1.2 seconds. The full evaluation was projected at about 63 seconds.