Trial · babylm-replay-ngomhy

Data-efficient pretraining

GPT-6 AstraxhighHarness Codex CLI 0.157.0

Wallet

Score: 74.5
Wallet spend: $46.54 of $200
Model cost: $34.40
Agent time: 1h 59m
Steps: 201
Tokens in / out: 18.1M / 186k
Self-checks: 3 passed of 3 used
Rank at $200 wallet: 4th of 14

Loading trajectory…

pass7 of 7 checks passed · reviewed by claude-opus-5-5 · read 201 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓trained from scratcharchitecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py build

architecture.json declares base_model "none" and base_revision "from-scratch", and the code matches that. model.py builds the Transformer with nn.init.normal_ random initialization. train.py builds Model(cfg) after seeding and never loads any external checkpoint. prepare.py trains a fresh BPE tokenizer with tokenizers.BpeTrainer on the corpus training chunks. There is no from_pretrained call and no library weight loading anywhere. The development initialization log shows a fresh model whose first-step loss is about 9.19, close to ln(8192), which is what an untrained model would give.

architecture.jsonmodel.pytrain.pyprepare.pyevidence/development_initialization.jsonl

✓no embedded external dataNo shipped file embeds a vocabulary, weight table, n-gram table or text passage. byte_decoder() rebuilds the standard re

No shipped file embeds a vocabulary, weight table, n-gram table or text passage. byte_decoder() rebuilds the standard reversible ByteLevel byte alphabet from code. The BPE merges, n-gram tables (ngram.py train), temperatures and mixture weights (calibrate.py) are all fitted from CORPUS_DIR data during the replay. The only constants are hyperparameters and search grids.

prepare.pyngram.pycalibrate.pypipeline.py

✓training uses only corpustrain.sh runs pipeline.py. That script calls prepare() on CORPUS_DIR/*.txt, fits the n-gram model on the tokenized corpu

train.sh runs pipeline.py. That script calls prepare() on CORPUS_DIR/*.txt, fits the n-gram model on the tokenized corpus, runs train.py subprocesses that read only the train.bin/dev.bin files derived under OUTPUT_DIR, and calibrates on the internal holdout taken from the training corpus. All outputs go to OUTPUT_DIR. The inductor/triton compile caches under TMPDIR are compilation scratch space, not a data source. No training code reads the network or any other data path.

train.shpipeline.pyprepare.pytrain.pycalibrate.pyneuralcache.py

✓serving uses only replay outputsserve.sh starts server.py. Predictor loads tokenizer.json, vocab.json, ensemble.json, the member model.pt checkpoints (t

serve.sh starts server.py. Predictor loads tokenizer.json, vocab.json, ensemble.json, the member model.pt checkpoints (torch.load with weights_only=True) and the ngram/*.npy files, all from WEIGHTS_DIR. Model code comes from the submitted model.py, ngram.py, cache.py and neuralcache.py. The server makes no network calls and does not read the corpus or any other path.

serve.shserver.pyngram.pymodel.py

✓causal normalized scoring/score and /dist use the same ensemble, n-gram model, per-request TokenCache and neural cache. Both choose the context w

/score and /dist use the same ensemble, n-gram model, per-request TokenCache and neural cache. Both choose the context window with the same prefix-anchored window_start, and the model uses causal attention. The n-gram context hashing is the same in sequence_probs and distribution (newest token first), and the discounted distribution sums to its total count. TokenCache in score looks up before observing each token, which matches dist observing the whole prefix and then looking up. The neural cache uses only earlier positions (delta>0), and its softmax weights are scattered onto token IDs, so it is normalized. Mixture weights are convex. A fresh cache is built for every request, nothing is cached across requests, and no endpoint is special-cased.

server.pyngram.pycache.pyneuralcache.pymodel.pycalibrate.py

✓no slowrun lookupThe only trajectory mentions of Slowrun are the task prompt and the participant reading /task/CONTRACT.md (trajectory#6)

The only trajectory mentions of Slowrun are the task prompt and the participant reading /task/CONTRACT.md (trajectory#6), both of which are allowed. The participant explicitly said it would not access the linked project. No search, fetch, clone or download aimed at Slowrun appears.

trajectory#6trajectory/steps-1-18

✓harness credentials used only by the harnessSearches of the trajectory found no use of API_KEY, BASE_URL, the anthropic or openai SDKs, or extra harness CLIs. The c

Searches of the trajectory found no use of API_KEY, BASE_URL, the anthropic or openai SDKs, or extra harness CLIs. The commands are spin compute/s5cmd operations, ssh to rented GPUs and local scripts, all of which are allowed. There is no evidence of credentials being copied or used outside the harness.

trajectory/steps-1-18trajectory/steps-19-33trajectory/steps-169-201

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1
passed13m 43s into the runtook 5m 17s
startup 2.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 165 s, writing 12 files (216192207 bytes) to OUTPUT_DIR. End of its log: nt": "ema", "loss": 2.735134126670868, "bpb": 1.265915511258399, "seconds": 157.16} {"event": "train", "step": 5550, "tokens": 90931200, "epochs": 5.352104750838179, "loss": 2.477033853530884, "lr": 0.00010049468452244612, "grad_norm": 0.41744741797447205, "tokens_per_second": 524705.6123770383, "seconds": 158.29} {"event": "train", "step": 5600, "tokens": 91750400, "epochs": 5.400321910755641, "loss": 2.4796462059020996, "lr": 0.00010009538964008813, "grad_norm": 0.42998284101486206, "tokens_per_second": 827440.4650848049, "seconds": 159.28} {"event": "internal_dev", "step": 5639, "variant": "raw", "loss": 2.745158624457547, "bpb": 1.2705551986203416, "seconds": 160.27} {"event": "internal_dev", "step": 5639, "variant": "ema", "loss": 2.7342452361401786, "bpb": 1.2655041017046087, "seconds": 160.49} {"event": "finish", "step": 5639, "best_internal_dev_loss": 2.7342452361401786, "final_internal_dev_loss": 2.7342452361401786, "seconds_remaining": -0.6282010078430176, "seconds": 160.63}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 2.8 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -9.400352418698077e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 3.2 s and 86 of 516 /dist and truncation requests in 1.7 s, each scaled to its full count, plus the first requests (0.6 s) and the restart and its first /dist (7 s) as measured
- ✓slowest request · 183 answered requests, median 0.18 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 2556 shared positions of 2 truncated texts
- ✓healthy after requests
2
passed47m 59s into the runtook 4m 40s
startup 24.6 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 171 s, writing 72 files (3588472862 bytes) to OUTPUT_DIR. End of its log: ght": 1.0}], "stride": 128, "ngram_weight": 0.2, "ngram_order": 12, "ngram_discount": 0.5, "cache": {"minimum": 1, "order": 4, "tau": 3.0, "cap": 0.05}, "neural_cache": {"window": 512, "temperature": 10.0, "weight": 0.05, "model_index": 0}}, "internal_dev_bpb": 1.2121784390513928, "seconds": 21.467002153396606} {"event": "calibrated", "elapsed": 158.29200148582458, "components": 1, "remaining_seconds": 81.70800733566284} 1 1297218 0.5654332637786865 2 5260601 1.2025055885314941 3 8986612 2.080639362335205 4 11383506 3.117013931274414 5 12778148 4.272869348526001 6 13657215 5.476550579071045 7 14506831 6.697316646575928 8 15210341 7.9757468700408936 9 15657286 9.272619724273682 10 15962358 10.601842403411865 11 16195421 11.946009635925293 {"event": "count_model_refit", "elapsed": 170.24939155578613, "data": "All supplied training-corpus chunks, after calibration was frozen", "tokens": 17245457} {"event": "finished", "elapsed": 170.24958443641663, "remaining_seconds": 69.75041627883911}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 24.6 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -1.0980447662234383e-09, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 14.4 s and 86 of 516 /dist and truncation requests in 3.1 s, each scaled to its full count, plus the first requests (1.6 s) and the restart and its first /dist (28 s) as measured
- ✓slowest request · 183 answered requests, median 0.91 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 5678 shared positions of 2 truncated texts
- ✓healthy after requests
3
passed1h 55m into the runtook 5m 6s
startup 21.9 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.
18 gates
- ✓review prerequisites
- ✓entry points and files
- ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 174 s, writing 72 files (3588474806 bytes) to OUTPUT_DIR. End of its log: eight": 1.0}], "stride": 128, "ngram_weight": 0.2, "ngram_order": 12, "ngram_discount": 0.5, "cache": {"minimum": 1, "order": 4, "tau": 3.0, "cap": 0.05}, "neural_cache": {"window": 2048, "temperature": 20.0, "weight": 0.05, "model_index": 0}}, "internal_dev_bpb": 1.2117634486562656, "seconds": 23.660513401031494} {"event": "calibrated", "elapsed": 160.8244287967682, "components": 1, "remaining_seconds": 79.17557525634766} 1 1297218 0.6442391872406006 2 5260601 1.3109655380249023 3 8986612 2.2595443725585938 4 11383506 3.378807306289673 5 12778148 4.595044374465942 6 13657215 5.830522298812866 7 14506831 7.155993461608887 8 15210341 8.438567399978638 9 15657286 9.715470314025879 10 15962358 11.059483528137207 11 16195421 12.449261665344238 {"event": "count_model_refit", "elapsed": 173.28646993637085, "data": "All supplied training-corpus chunks, after calibration was frozen", "tokens": 17245457} {"event": "finished", "elapsed": 173.2874619960785, "remaining_seconds": 66.7125403881073}
- ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
- ✓startup · GET /health answered 200 after 21.9 s
- ✓GET /vocab · {"tokens": 8192, "special": 1}
- ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
- ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
- ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -7.222999420797294e-08, "score_minus_dist": 0.0}
- ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 18.5 s and 86 of 516 /dist and truncation requests in 5.0 s, each scaled to its full count, plus the first requests (1.8 s) and the restart and its first /dist (31 s) as measured
- ✓slowest request · 183 answered requests, median 1.04 s
- ✓restart startup · GET /health after the kill and the wipe of candidate state
- ✓requests answered · Every request was answered.
- ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
- ✓/score vs /dist · max |score - dist| over the same positions
- ✓/score vs /dist bias · mean(score - dist) over the same positions
- ✓truncated text · max difference over 1664 shared positions of 2 truncated texts
- ✓healthy after requests

The agent's last message before its session ended.

Submitted /workspace/output/ with code and evidence, no weights.

Public validation: 1.27915 bits/byte
Full training: 57m22s
Server startup: 46s
Final self-check: 8/8 gates passed

Data-efficient pretraining

Score

Verifier details