Data-efficient pretraining

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1. 1
   failed5h 33m into the runtook 2m 52s

   failed train.sh: With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: manager(key) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/triton/runtime/cache.py", line 246, in get_cache_manager return cls(_base32(key)) ^^^^^^^^^^^^^^^^^

   3 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✗train.sh · With TRAIN_BUDGET_SECONDS=240: train.sh exited with status 1: manager(key) ^^^^^^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/triton/runtime/cache.py", line 246, in get_cache_manager return cls(_base32(key)) ^^^^^^^^^^^^^^^^^ File "/opt/conda/lib/python3.11/site-packages/triton/runtime/cache.py", line 53, in __init__ os.makedirs(self.cache_dir, exist_ok=True) File "<frozen os>", line 225, in makedirs torch._inductor.exc.InductorError: PermissionError: [Errno 13] Permission denied: '/replay/tmp/trainwork.190/triton/PGF237LIWRH5N5P4JFR6NWPMDMVV36YIKZDUPAVTP65TCPKWKU7A' Set TORCHDYNAMO_VERBOSE=1 for the internal stack trace (please do this especially if you're reporting a bug to PyTorch). For even more developer context, set TORCH_LOGS="+dynamo" [train.sh] no meta.json produced; train.py status 1
2. 2
   passed5h 41m into the runtook 6m 32s

   train.sh, given TRAIN_BUDGET_SECONDS=240, was still running when the self-check stopped it after 309 s, so serve.sh, the restart, the speed gates and the audits were not exercised; a train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them.

   3 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, still running without error on NVIDIA H100 80GB HBM3 after 309 s; stopped there, so serve.sh, the startup deadline, the evaluation deadlines and the audits were not exercised. A train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them. End of its log: :19:08] open_subtitles.train.txt: 386046 lines sha256=9b21adecf57d [prepare 08:19:08] simple_wiki.train.txt: 58709 lines sha256=0ac614aa069e [prepare 08:19:08] switchboard.train.txt: 2892 lines sha256=89c99649777b [prepare 08:19:08] docs[bnc_spoken.train.txt]: 388 [prepare 08:19:08] docs[childes.train.txt]: 1579 [prepare 08:19:08] docs[gutenberg.train.txt]: 1197 [prepare 08:19:08] docs[open_subtitles.train.txt]: 1281 [prepare 08:19:08] docs[simple_wiki.train.txt]: 746 [prepare 08:19:08] docs[switchboard.train.txt]: 12 [prepare 08:19:08] train docs: 5203, 54.4 MB [prepare 08:19:12] tokenizer trained in 3.7s, vocab=8192 [prepare 08:19:12] prepare: 3.8s so far [prepare 08:19:17] tokenized 5203 docs in 5.1s (17.23M tokens, 3372K tok/s) [prepare 08:19:18] {"vocab_size": 8192, "train_docs": 5203, "train_tokens": 17232166, "train_bytes": 54391141, "dev_docs": 0, "prepare_seconds": 9.68531346321106} [train 08:19:21] data: 5203 docs, 17.23M tokens; buckets [1024, 2048, 3072, 4096]; dev: 0 docs
3. 3
   passed6h 36m into the runtook 2m 26s

   startup 3.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.

   18 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 51 s, writing 9 files (69087184 bytes) to OUTPUT_DIR. End of its log: .0, 24.0, 18.0, 24.0, 24.0] epochs), budget 0.1 min, 94.4 ms/step [train 09:15:02] member 0: 300 steps (0.6 epochs) [train 09:15:02] m0 ema decay 0.966667 (horizon 30 steps) [train 09:15:07] m0 step 50/300 loss 6.5335 lr 5.73e-03 299.1K tok/s 100 ms/step [train 09:15:12] m0 step 100/300 loss 5.5041 lr 4.73e-03 316.5K tok/s 94 ms/step [train 09:15:16] m0 step 150/300 loss 4.9761 lr 3.29e-03 319.0K tok/s 94 ms/step [train 09:15:21] m0 step 200/300 loss 4.8079 lr 1.81e-03 317.2K tok/s 94 ms/step [train 09:15:26] m0 step 250/300 loss 4.5374 lr 7.05e-04 316.1K tok/s 95 ms/step [train 09:15:30] m0 step 300/300 loss 4.5737 lr 3.00e-04 315.3K tok/s 94 ms/step [train 09:15:31] m0 saved model_0.safetensors (ema) in 0.5 min, best dev ema None [train 09:15:31] deadline reached; skipping remaining members [train 09:15:31] skipping n-gram build (remaining budget -20s) [train 09:15:31] training finished with 1/6 members in 0.6 min [train.sh] done: 1 ensemble member(s), train.py status 0, elapsed 51s
   • ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
   • ✓startup · GET /health answered 200 after 3.4 s
   • ✓GET /vocab · {"tokens": 8192, "special": 0}
   • ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1436}
   • ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 983}
   • ✓POST /dist · {"prefix_tokens": 718, "logprobs": 8192, "logsumexp": -2.6614237036426402e-08, "score_minus_dist": 0.0}
   • ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 1.7 s and 86 of 516 /dist and truncation requests in 1.3 s, each scaled to its full count, plus the first requests (0.3 s) and the restart and its first /dist (8 s) as measured
   • ✓slowest request · 183 answered requests, median 0.11 s
   • ✓restart startup · GET /health after the kill and the wipe of candidate state
   • ✓requests answered · Every request was answered.
   • ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
   • ✓/score vs /dist · max |score - dist| over the same positions
   • ✓/score vs /dist bias · mean(score - dist) over the same positions
   • ✓truncated text · max difference over 963 shared positions of 2 truncated texts
   • ✓healthy after requests

The agent's last message before its session ended.