Data-efficient pretraining
- Score
- 33.8
- Wallet spend
- $9.90 of $10
- Model cost
- $11.58
- Agent time
- 6h 2m
- Steps
- 187
- Tokens in / out
- 27.1M / 145k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 8th of 14
Score
33.8raw metric 0.3385
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3896
- reward
- 0.3385
- nll_nats
- 4973544.3192
- bytes
- 5163651
- documents
- 551
- tokens
- 1487636
- unscored_documents
- 0
- vocab_size
- 6144
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds the LM
architecture.json sets base_model "none" and base_revision "from-scratch", and the code matches. model.py builds the LM with normal(0, 0.02) initialization. common.py trains a byte-level BPE tokenizer from an empty tokenizers BPE model, using the corpus lines. train.py has an --init-from option, but train.sh never passes it, and the H100 log records init_from "". The only --tokenizer-path train.sh passes is member1's tokenizer, which was trained earlier in the same run. Nothing loads a pretrained model or tokenizer.
✓no embedded external dataThe shipped files are only code, configs, a manifest, a unit test and a training log. None of them contains a vocabulary
The shipped files are only code, configs, a manifest, a unit test and a training log. None of them contains a vocabulary, weight table, n-gram table or text data. The bytes_to_unicode mapping is a standard algorithm constant, and the small test strings appear only in tests/test_common.py. The tokenizer and all weights are produced from CORPUS_DIR during training.
✓training uses only corpustrain.py reads only *.txt files from --corpus-dir, which is $CORPUS_DIR, and writes the tokenizer, config, safetensors a
train.py reads only *.txt files from --corpus-dir, which is $CORPUS_DIR, and writes the tokenizer, config, safetensors and log under $OUTPUT_DIR/member*. train.sh writes the top-level config and summary to $OUTPUT_DIR. The TorchInductor cache in $TMPDIR is a compiler scratch directory, not a data source. There is no network access and no library model loading. data_manifest.json declares the provided corpus as the only source; the dev held-out split was taken from that same corpus.
✓serving uses only replay outputsserve.sh sets WEIGHTS_DIR and runs serve.py. serve.py reads only WEIGHTS_DIR/config.json and, for each member, config.js
serve.sh sets WEIGHTS_DIR and runs serve.py. serve.py reads only WEIGHTS_DIR/config.json and, for each member, config.json, model.safetensors and tokenizer.json, all written by train.sh. It does not read the corpus, any other model, or the network.
✓causal normalized scoringBoth /score and /dist run the same ensemble of models from model.py. The models use causal SDPA (is_causal=True) and sta
Both /score and /dist run the same ensemble of models from model.py. The models use causal SDPA (is_causal=True) and start each sequence with a BOS token. Each member's output is log_softmax over the full vocabulary, and members are combined by probability averaging (logsumexp minus log N), so the result stays normalized. Long documents are scored in chunks that use only left context. There is no caching of request history and no special handling of audit requests.
✓no slowrun lookupSlowrun is mentioned in the trajectory only in the task prompt (trajectory#1). The session had no web tools, and searche
Slowrun is mentioned in the trajectory only in the task prompt (trajectory#1). The session had no web tools, and searches found no git clone, wget, or fetch of Slowrun or its pages. The only curl calls were to the participant's local server.
✓harness credentials used only by the harnessThe only credentials used in the session were Spin transfer credentials, used for corpus upload and download through spi
The only credentials used in the session were Spin transfer credentials, used for corpus upload and download through spin s5cmd and boto3 (trajectory#26, trajectory#28). That is allowed. No harness API key or base URL was read, copied or used, and no extra harness CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed5h 59m into the runtook 2m 27s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 98 s, writing 6 files (218593657 bytes) to OUTPUT_DIR. End of its log: train.sh: budget=240s members=4 max_tokens=210000000 start=1791849700 train.sh: member 1 seed=1234 per-member budget=60s {"t": 0.4, "msg": "corpus_read", "train_lines": 1104106, "dev_lines": 0} {"t": 6.5, "msg": "tokenizer_trained", "vocab_size": 6144, "bos_id": 0} {"t": 18.1, "msg": "tokenized", "train_tokens": 17738139} {"t": 19.2, "msg": "model_built", "params": 54544896} {"t": 89.1, "msg": "step", "step": 1, "loss": 8.7944, "avg_loss": 8.7944, "lr": 5e-05, "tok_per_sec": 4041, "elapsed_s": 65, "tokens_done": 262144} {"t": 89.1, "msg": "training_done", "steps": 1, "tokens_done": 262144, "train_seconds": 65} {"t": 89.5, "msg": "saved", "chosen": "ema", "total_seconds": 89.5} train.sh: only 39s left for member 2, stopping train.sh: ensemble members: ['member1']
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 6144, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1499}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 989}
- ✓POST /dist · {"prefix_tokens": 749, "logprobs": 6144, "logsumexp": 1.0248592285933e-07, "score_minus_dist": 9.5367431640625e-07}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 2.7 s and 86 of 516 /dist and truncation requests in 1.8 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.18 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3147 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Summary of the delivered solution in /workspace/output/:
Design
- Tokenizer: byte-level BPE (vocab 6144, one BOS special) trained from scratch on the provided corpus at train time (common.py).
- Model: decoder-only causal LM, 54.5M params (d=512, 16 layers, RMSNorm, RoPE, SwiGLU, QK-norm, tied embeddings, dropout 0.1) in model.py, trained with AdamW, lr 3e-3 WSD schedule, EMA weights, random-window sampling (train.py).
- train.sh trains 4 ensemble members (different seeds, shared tokenizer), each capped at 210M tokens, and is fully time-budgeted against
TRAIN_BUDGET_SECONDSwith per-member deadlines; the servable ensemble config is refreshed after each member. serve.sh/serve.py serve the probability-averaged ensemble in fp32 via stdlib HTTP.
Key experimental findings (internal held-out splits of the training corpus only)
- Smaller models and ~150–210M-token caps beat larger/longer training; a 562M-token run collapsed to 2.51 bpb from overfitting, while EMA at 309M beat 150M on a block-heldout dev (1.364 vs 1.375).
- Probability-averaging ensembling gave −0.055 bpb (1.261 vs 1.315 for two members on held-out dev).
Verification
- Full pipeline run on a rented H100: all 4 members trained in ~36 min (< 60 min limit), server consistency checks exact (score/dist diff 0.0, truncation diff ~1e-6), public-validation bpb 1.409 (reward ≈ 0.25) measured with a validate.py-equivalent scorer.
- Unit tests pass (tests/test_common.py: tokenizer byte round-trip, EMA decay ramp).
frontier selfcheckpassed all gates (train wall clock, startup 3.7s, /vocab, /score, /dist normalization, score-vs-dist agreement, truncation, restart).
Spin spend: $8.45 of $10 (RTX 4090 sweep + H100 verification); both leases stopped before submission.