Data-efficient pretraining
- Score
- 77.8
- Wallet spend
- $20.94 of $50
- Model cost
- $51.04
- Agent time
- 2h 17m
- Steps
- 235
- Tokens in / out
- 29.5M / 262k
- Self-checks
- 4 passed of 4 used
- Rank at $50 wallet
- 2nd of 14
Score
77.8raw metric 0.7777
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.2969
- reward
- 0.7777
- nll_nats
- 4641817.4955
- bytes
- 5163651
- documents
- 551
- tokens
- 1427885
- unscored_documents
- 0
- vocab_size
- 8192
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch". The code agrees. train.py builds Language
architecture.json declares base_model "none" and base_revision "from-scratch". The code agrees. train.py builds LanguageModel(c).cuda() from a random initialization; model.py uses normal init with std 0.02. No code anywhere loads pretrained weights. The tokenizer is a byte-level BPE that data.py trains on the corpus. An rg check in the trajectory found no from_pretrained or huggingface calls in the output code.
✓no embedded external dataNo weights or external data tables were shipped; there is no weight inventory and the trajectory confirms no weights wer
No weights or external data tables were shipped; there is no weight inventory and the trajectory confirms no weights were shipped. ngram.py builds its n-gram tables at train time from train_tokens.npy, and calibrate.py fits its calibration on the internal holdout. recipe.json and optim.py hold only hyperparameters and algorithm constants. architecture.json lists external_data as None and downloads as an empty list.
✓training uses only corpustrain.sh runs pipeline.py. pipeline.py calls data.prepare on CORPUS_DIR only, then builds the n-gram tables from those t
train.sh runs pipeline.py. pipeline.py calls data.prepare on CORPUS_DIR only, then builds the n-gram tables from those tokens, then trains the members with train.py, then calibrates with calibrate.py on an internal 2% SHA256 holdout taken from the corpus. architecture.json data_provenance names the six corpus train files. In the trajectory, the corpus moved to the dev machine by tar and s5cmd from /task/public/corpus. The only other downloads were pip packages and a cloudflared binary for a dev tunnel; none are training data. validation.jsonl was not used for training or calibration.
✓serving uses only replay outputsserve.sh runs server.py. Scorer(WEIGHTS_DIR) loads only files produced by train.sh: tokenizer.json, vocab.json, ensemble
serve.sh runs server.py. Scorer(WEIGHTS_DIR) loads only files produced by train.sh: tokenizer.json, vocab.json, ensemble.json, the member config.json and model.pt files (torch.load with weights_only=True), calibration.json, and the ngram/order*.npz tables. Serving makes no network calls and loads no external models.
✓causal normalized scoringinference.py scores with sliding windows and causal SDPA attention (is_causal=True in model.py). The n-gram lookup in ng
inference.py scores with sliding windows and causal SDPA attention (is_causal=True in model.py). The n-gram lookup in ngram.py and the context cache in context_cache.py use only the prefix, and the cache is created fresh for each request, so nothing persists across requests. In server.py, /dist mixes the distributions as convex mixtures over the full vocabulary and normalizes with logaddexp. /score uses the same mixture evaluated at the target token. The self-check gates in the trajectory measured /dist normalization error and /score-vs-/dist differences of 0.0.
✓no slowrun lookupSlowrun is mentioned only in the task prompt and in the participant's notes saying not to access it. No step in the traj
Slowrun is mentioned only in the task prompt and in the participant's notes saying not to access it. No step in the trajectory searched for, fetched, cloned or opened qlabs-eng/slowrun or pages about it.
✓harness credentials used only by the harnessThe trajectory shows OPENAI_API_KEY only as a variable name in an env listing restricted to variable names (env | cut -d
The trajectory shows OPENAI_API_KEY only as a variable name in an env listing restricted to variable names (env | cut -d= -f1). No submission or session code calls model APIs with the harness keys or base URL, copies or echoes them, or launches extra harness CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed7m 28s into the runtook 5m 8s
startup 3.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 146 s, writing 12 files (254516018 bytes) to OUTPUT_DIR. End of its log: _per_second": 991745, "elapsed": 141.35} {"event": "train", "step": 2760, "loss": 2.4672067165374756, "lr": 8.053449556775781e-05, "grad_norm": 0.34119707345962524, "tokens_per_second": 994639, "elapsed": 142.01} {"event": "train", "step": 2780, "loss": 2.4678757190704346, "lr": 8.015294064745701e-05, "grad_norm": 0.33901169896125793, "tokens_per_second": 979749, "elapsed": 142.68} {"event": "train", "step": 2800, "loss": 2.4618887901306152, "lr": 8.000260215654168e-05, "grad_norm": 0.3513486683368683, "tokens_per_second": 986028, "elapsed": 143.34} {"event": "evaluation", "step": 2800, "raw_bpb": 1.282960057258606, "ema_bpb": 1.2808058261871338, "best_bpb": 1.2808058261871338, "best_step": 2800, "elapsed": 144.08} {"event": "evaluation", "step": 2802, "raw_bpb": 1.2828733921051025, "ema_bpb": 1.2807762622833252, "best_bpb": 1.2807762622833252, "best_step": 2802, "elapsed": 144.9} {"event": "finished", "step": 2802, "best_step": 2802, "best_bpb": 1.2807762622833252, "elapsed": 144.94}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.7 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -4.383709417687953e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 5.5 s and 86 of 516 /dist and truncation requests in 1.2 s, each scaled to its full count, plus the first requests (0.5 s) and the restart and its first /dist (8 s) as measured
- ✓slowest request · 183 answered requests, median 0.25 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2730 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed38m 12s into the runtook 3m 45s
startup 4.7 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 154 s, writing 21 files (918128230 bytes) to OUTPUT_DIR. End of its log: 120, "elapsed": 99.62} {"event": "finished", "step": 120, "best_step": 120, "best_bpb": 1.9126757383346558, "elapsed": 99.63} {"event": "member_finished", "elapsed": 147.651, "member": 0, "summary": {"steps": 120, "best_step": 120, "best_internal_dev_bpb": 1.9126757383346558, "elapsed_seconds": 99.62621474266052}} {"event": "calibration", "temperature": 1.0, "corpus_ngram": {"weight": 0.2, "min_order": 3, "tau": 0.0}, "context_cache": {"weight": 0.2, "min_order": 2, "tau": 0.0}, "member_weights": [1.0], "member_temperatures": [0.95], "internal_holdout_bpb": 1.74591913623715, "calibrated_neural_bpb": 1.8665067978179142, "source": "Only deterministic internal holdout from the supplied CORPUS_DIR training files", "holdout_bytes": 49280, "collection_seconds": 3.773195505142212, "elapsed_seconds": 3.9318058490753174} {"event": "calibration_finished", "elapsed": 153.804, "returncode": 0} {"event": "pipeline_finished", "elapsed": 153.805, "members": 1, "seconds_remaining": 86.19518947601318}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 4.7 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 2.220446049250313e-16, "score_minus_dist": -5.240202938239236e-09}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 11.8 s and 86 of 516 /dist and truncation requests in 3.0 s, each scaled to its full count, plus the first requests (0.4 s) and the restart and its first /dist (10 s) as measured
- ✓slowest request · 183 answered requests, median 0.38 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1512 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed1h 15m into the runtook 4m 36s
startup 8.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 184 s, writing 25 files (3566221830 bytes) to OUTPUT_DIR. End of its log: ep": 18, "best_internal_dev_bpb": 2.7825675010681152, "elapsed_seconds": 87.50253176689148}} {"event": "calibration", "temperature": 1.0, "corpus_ngram": {"weight": 1.0, "min_order": 3, "tau": 0.0, "weights_by_order": [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.15, 0.4], [0.15, 0.3], [0.15, 0.3], [0.2, 0.3], [0.2, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]}, "context_cache": {"weight": 0.2, "min_order": 2, "tau": 0.0}, "member_weights": [1.0], "member_temperatures": [0.9], "internal_holdout_bpb": 2.3260062680691487, "calibrated_neural_bpb": 2.691640986337279, "neural_and_ngram_bpb": 2.4396125124328267, "source": "Only deterministic internal holdout from the supplied CORPUS_DIR training files", "holdout_bytes": 49280, "collection_seconds": 7.147217750549316, "elapsed_seconds": 7.34001350402832} {"event": "calibration_finished", "elapsed": 185.016, "returncode": 0} {"event": "pipeline_finished", "elapsed": 185.017, "members": 1, "seconds_remaining": 54.98310160636902}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 8.5 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 8.881784197001252e-16, "score_minus_dist": 9.08169059954389e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 13.5 s and 86 of 516 /dist and truncation requests in 4.2 s, each scaled to its full count, plus the first requests (1.0 s) and the restart and its first /dist (14 s) as measured
- ✓slowest request · 183 answered requests, median 0.58 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5256 shared positions of 2 truncated texts
- ✓healthy after requests
- 4passed2h 13m into the runtook 4m 20s
startup 10.5 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 178 s, writing 25 files (3566219819 bytes) to OUTPUT_DIR. End of its log: st_step": 1, "best_internal_dev_bpb": 4.068833827972412, "elapsed_seconds": 88.11288785934448}} {"event": "calibration", "temperature": 1.0, "corpus_ngram": {"weight": 1.0, "min_order": 3, "tau": 0.0, "weights_by_order": [[0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.15, 0.5], [0.2, 0.4], [0.2, 0.4], [0.3, 0.4], [0.3, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0], [0.0, 0.0]]}, "context_cache": {"weight": 0.2, "min_order": 2, "tau": 0.0}, "member_weights": [1.0], "member_temperatures": [1.2], "internal_holdout_bpb": 2.79640340355035, "calibrated_neural_bpb": 3.5624832776640676, "neural_and_ngram_bpb": 2.975465365683118, "source": "Only deterministic internal holdout from the supplied CORPUS_DIR training files", "holdout_bytes": 49280, "collection_seconds": 8.506871700286865, "elapsed_seconds": 8.706109762191772} {"event": "calibration_finished", "elapsed": 178.238, "returncode": 0} {"event": "pipeline_finished", "elapsed": 178.239, "members": 1, "seconds_remaining": 61.76145815849304}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 10.5 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": 0.0, "score_minus_dist": -7.513795985403249e-08}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 14.9 s and 86 of 516 /dist and truncation requests in 4.3 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (15 s) as measured
- ✓slowest request · 183 answered requests, median 0.69 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 4585 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted source and logs in /workspace/output/, including train.sh and serve.sh.
- Full training: 56m 2s
- Server startup: 13.4s
- Public validation: 1.27355 bits/byte
- Final self-check: all 8 gates passed
No weights shipped.