Data-efficient pretraining
- Score
- 59.5
- Wallet spend
- $8.73 of $10
- Model cost
- $42.48
- Agent time
- 2h 11m
- Steps
- 281
- Tokens in / out
- 27.5M / 193k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 2nd of 14
Score
59.5raw metric 0.5951
The raw metric (bits per byte on held-out text) is rescaled so that 1.461 bits per byte, the organizers' 7-minute GPT-2 baseline scores 0 and 1.25 bits per byte, a target set by the task author scores 100, clamped to that range.
Verifier details
- metric
- bits_per_utf8_byte
- bpb
- 1.3354
- reward
- 0.5951
- nll_nats
- 4779734.0950
- bytes
- 5163651
- documents
- 551
- tokens
- 1427908
- unscored_documents
- 0
- vocab_size
- 8256
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py sets
architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py sets up every Linear and Embedding layer with nn.init.normal_. train.py builds LanguageModel(config) from a seed and never loads a checkpoint. data.py trains a new BPE tokenizer with tokenizers.Tokenizer(models.BPE()).train_from_iterator on the corpus text. Nothing calls from_pretrained or loads a pretrained tokenizer or model. The trajectory shows a code-only submission whose weights come from train.sh.
✓no embedded external dataThe shipped files are code, recipe.json (hyperparameters only), architecture.json and logs/evidence documents. The only
The shipped files are code, recipe.json (hyperparameters only), architecture.json and logs/evidence documents. The only constants are algorithm constants: the GPT-2 byte-level alphabet mapping in data.py, the Newton-Schulz coefficients in optim.py, RoPE frequencies, KN discount formulas and the temperature grid. The vocabulary, merge lists, n-gram tables, weights and calibration values are all computed during the replay from CORPUS_DIR.
✓training uses only corpusdata.prepare reads only sorted(Path(CORPUS_DIR).glob('*.txt')) and writes the tokenizer, vocab, manifest and token strea
data.prepare reads only sorted(Path(CORPUS_DIR).glob('*.txt')) and writes the tokenizer, vocab, manifest and token streams to OUTPUT_DIR. orchestrate.py, train.py, ngram.fit and calibrate.py read and write only under OUTPUT_DIR. The selection and calibration reserve is a deterministic hash-based holdout of the supplied training chunks. The code makes no network calls. The only other paths are the inductor and triton compile caches in TMPDIR, which are compiler scratch space, not data. The corpus source is declared in architecture.json, and the trajectory shows the supplied corpus files being transferred unchanged to the rented machine.
✓serving uses only replay outputsserve.sh runs serve.py, which reads only WEIGHTS_DIR: tokenizer.json, vocab.json, ensemble.json, each model_*/config.jso
serve.sh runs serve.py, which reads only WEIGHTS_DIR: tokenizer.json, vocab.json, ensemble.json, each model_*/config.json and model.pt, and the ngram/*.npy tables written by orchestrate.py and ngram.fit. It reads no corpus, cache or network resource.
✓causal normalized scoringBoth /score and /dist use the same Member.forward. The model uses causal SDPA, a BOS-prefixed window with a stride of co
Both /score and /dist use the same Member.forward. The model uses causal SDPA, a BOS-prefixed window with a stride of context/2, and the same window start in both paths (score uses max(0,block-stride); dist uses (i//stride-1)*stride). The output is a log_softmax over the full padded vocabulary, with special tokens masked to -10000. The ensemble is mixed with logaddexp using fixed weights. The interpolated KN n-gram uses only history tokens, its distribution is normalized, and the gated confidence computes the same value in score and dist. There is no request caching or history state: the input buffer is zeroed on each call, and the lock only serializes requests.
✓no slowrun lookupThe only mentions of Slowrun in the trajectory are the task prompt (steps 1-18) and a read of /task/CONTRACT.md at traje
The only mentions of Slowrun in the trajectory are the task prompt (steps 1-18) and a read of /task/CONTRACT.md at trajectory#121, which is allowed. No search, fetch, clone or download aimed at Slowrun appears.
✓harness credentials used only by the harnessSearches for API_KEY, BASE_URL, OPENAI, printenv and other harness CLIs found nothing. The S3 credentials in transfer.js
Searches for API_KEY, BASE_URL, OPENAI, printenv and other harness CLIs found nothing. The S3 credentials in transfer.json come from the Spin transfer facility and were used only to move the supplied corpus (trajectory#55); they are not harness credentials. The trajectory shows no model API calls from the participant's own code and no extra harness sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed5m 10s into the runtook 6m 5s
startup 3.3 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 231 s, writing 11 files (215793472 bytes) to OUTPUT_DIR. End of its log: "elapsed": 225.93, "loss": 2.19953, "lr": 5.1740946279708904e-05, "grad_norm": 0.3209, "tokens_per_second": 898315, "allocated_gb": 5.81} {"event": "train", "step": 9350, "tokens": 153190400, "elapsed": 226.84, "loss": 2.20498, "lr": 4.9094145209077836e-05, "grad_norm": 0.3303, "tokens_per_second": 898902, "allocated_gb": 5.81} {"event": "train", "step": 9400, "tokens": 154009600, "elapsed": 227.75, "loss": 2.18055, "lr": 4.710375159116937e-05, "grad_norm": 0.3269, "tokens_per_second": 898822, "allocated_gb": 5.81} {"event": "train", "step": 9450, "tokens": 154828800, "elapsed": 228.66, "loss": 2.22017, "lr": 4.576887763149157e-05, "grad_norm": 0.3516, "tokens_per_second": 899053, "allocated_gb": 5.81} {"event": "train", "step": 9500, "tokens": 155648000, "elapsed": 229.57, "loss": 2.18221, "lr": 4.509218595491166e-05, "grad_norm": 0.3466, "tokens_per_second": 899543, "allocated_gb": 5.81} {"event": "complete", "steps": 9526, "tokens": 156073984, "elapsed_seconds": 230.35646319389343}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.3 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8256, "logsumexp": -1.204597088744208e-07, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.9 s and 86 of 516 /dist and truncation requests in 2.0 s, each scaled to its full count, plus the first requests (0.7 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.44 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3046 shared positions of 2 truncated texts
- ✓healthy after requests
- 2passed38m 19s into the runtook 4m 40s
startup 3.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 203 s, writing 59 files (1475353040 bytes) to OUTPUT_DIR. End of its log: : 5.75} {"event": "train", "step": 4300, "tokens": 70451200, "elapsed": 87.43, "loss": 2.69283, "lr": 0.0014112602402497913, "grad_norm": 0.2612, "tokens_per_second": 898264, "allocated_gb": 5.75} {"event": "checkpoint_selected", "step": 1527, "internal_loss": 3.1683792918920517} {"event": "complete", "steps": 4341, "tokens": 71122944, "elapsed_seconds": 88.58014464378357} {"event": "calibration_member", "path": "model_00"} {"event": "calibration_member", "path": "model_01"} {"event": "calibration_complete", "nll": 2.9983236292996054, "models": [{"path": "model_00", "weight": 0.49903041957870353}, {"path": "model_01", "weight": 0.5009695804212964}], "temperature": 0.95, "ngram_weight": 0.2, "ngram_gated": false} {"event": "replay_complete", "elapsed_seconds": 203.322744846344, "models": 2, "distribution": {"models": [{"path": "model_00", "weight": 0.49903041957870353}, {"path": "model_01", "weight": 0.5009695804212964}], "temperature": 0.95, "ngram_weight": 0.2, "ngram_gated": false}}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.8 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8256, "logsumexp": -4.3635003832420693e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 21.5 s and 86 of 516 /dist and truncation requests in 2.3 s, each scaled to its full count, plus the first requests (0.5 s) and the restart and its first /dist (9 s) as measured
- ✓slowest request · 183 answered requests, median 0.38 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 3558 shared positions of 2 truncated texts
- ✓healthy after requests
- 3passed2h 7m into the runtook 4m 20s
startup 3.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 199 s, writing 44 files (1409051088 bytes) to OUTPUT_DIR. End of its log: ull, "ema_loss": 3.276921361684799, "ema_variant": "short", "short_ema_loss": 3.276921361684799, "long_ema_loss": 3.3184076100587845, "source": "held-out chunks of supplied training corpus only"} {"event": "checkpoint_selected", "step": 1300, "internal_loss": 3.276921361684799} {"event": "complete", "steps": 1300, "tokens": 21299200, "elapsed_seconds": 52.84815311431885} {"event": "calibration_member", "path": "model_00"} {"event": "calibration_member", "path": "model_01"} {"event": "calibration_complete", "nll": 2.9659137513533134, "models": [{"path": "model_00", "weight": 0.8423168739520351}, {"path": "model_01", "weight": 0.15768312604796483}], "temperature": 1.0, "ngram_weight": 0.15, "ngram_gated": false} {"event": "replay_complete", "elapsed_seconds": 198.16465401649475, "models": 2, "distribution": {"models": [{"path": "model_00", "weight": 0.8423168739520351}, {"path": "model_01", "weight": 0.15768312604796483}], "temperature": 1.0, "ngram_weight": 0.15, "ngram_gated": false}}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 3.9 s
- ✓GET /vocab · {"tokens": 8256, "special": 64}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8256, "logsumexp": -3.517304625333395e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 9.1 s and 86 of 516 /dist and truncation requests in 1.4 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.22 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2159 shared positions of 2 truncated texts
- ✓healthy after requests
The agent's last message before its session ended.
Submitted code-only solution in /workspace/output/.
- Full H100 training: 57m 58s
- Server startup: 13.5s
- Public validation: 1.3498 bits/byte
- Final
frontier selfcheck: 8/8 gates passed