Data-efficient pretraining

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1. 1
   passed2h 19m into the runtook 5m 37s

   train.sh, given TRAIN_BUDGET_SECONDS=240, was still running when the self-check stopped it after 302 s, so serve.sh, the restart, the speed gates and the audits were not exercised; a train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them.

   3 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, still running without error on NVIDIA H100 80GB HBM3 after 302 s; stopped there, so serve.sh, the startup deadline, the evaluation deadlines and the audits were not exercised. A train.sh that sizes its run from TRAIN_BUDGET_SECONDS exits in time for them. End of its log: 6246 epochs=3.67 elapsed=153s [model 0] step 1100 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=3.84 elapsed=160s [model 0] step 1150 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.02 elapsed=168s [model 0] step 1200 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.19 elapsed=175s [model 0] step 1250 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.36 elapsed=182s [model 0] step 1300 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.54 elapsed=190s [model 0] step 1350 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.71 elapsed=197s [model 0] step 1400 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=4.89 elapsed=204s [model 0] step 1450 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=5.06 elapsed=211s [model 0] step 1500 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=5.24 elapsed=219s [model 0] step 1550 p=0.000 T=4096 B=16 lr=0.000 loss=7.6246 ema=7.6246 epochs=5.41 elapsed=226s
2. 2
   passed2h 55m into the runtook 5m 37s

   startup 5.4 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.

   18 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 230 s, writing 5 files (352489057 bytes) to OUTPUT_DIR. End of its log: d=38s [model 0] step 300 p=0.399 T=4096 B=16 lr=1.000 loss=2.9554 ema=5.6681 epochs=1.05 elapsed=46s [model 0] step 350 p=0.466 T=4096 B=16 lr=1.000 loss=2.7985 ema=5.3811 epochs=1.22 elapsed=53s [model 0] step 400 p=0.533 T=4096 B=16 lr=0.935 loss=2.7237 ema=5.1154 epochs=1.40 elapsed=61s [model 0] step 450 p=0.600 T=4096 B=16 lr=0.801 loss=2.8401 ema=4.8879 epochs=1.57 elapsed=68s [model 0] step 500 p=0.666 T=4096 B=16 lr=0.667 loss=2.6994 ema=4.6690 epochs=1.75 elapsed=76s [model 0] step 550 p=0.733 T=4096 B=16 lr=0.533 loss=2.5320 ema=4.4553 epochs=1.92 elapsed=83s [model 0] step 600 p=0.800 T=4096 B=16 lr=0.400 loss=2.6773 ema=4.2775 epochs=2.10 elapsed=91s [model 0] step 650 p=0.867 T=4096 B=16 lr=0.266 loss=2.6763 ema=4.1174 epochs=2.27 elapsed=99s [model 0] step 700 p=0.934 T=4096 B=16 lr=0.132 loss=2.3377 ema=3.9394 epochs=2.44 elapsed=106s [model 0] done steps=749 epochs=2.62 train_time=196s compile status: {'ns': True, 'model': True} total wall time since replay start: 228s
   • ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
   • ✓startup · GET /health answered 200 after 5.4 s
   • ✓GET /vocab · {"tokens": 2048, "special": 1}
   • ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1957}
   • ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 997}
   • ✓POST /dist · {"prefix_tokens": 978, "logprobs": 2048, "logsumexp": 0.0, "score_minus_dist": 3.916039099083335e-05}
   • ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 28.4 s and 86 of 516 /dist and truncation requests in 10.5 s, each scaled to its full count, plus the first requests (1.1 s) and the restart and its first /dist (11 s) as measured
   • ✓slowest request · 183 answered requests, median 1.39 s
   • ✓restart startup · GET /health after the kill and the wipe of candidate state
   • ✓requests answered · Every request was answered.
   • ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
   • ✓/score vs /dist · max |score - dist| over the same positions
   • ✓/score vs /dist bias · mean(score - dist) over the same positions
   • ✓truncated text · max difference over 4261 shared positions of 2 truncated texts
   • ✓healthy after requests
3. 3
   passed4h 56m into the runtook 5m 40s

   startup 4.8 s of 300 s; 8/8 gates passed; The evaluation deadline was projected from a sample: 91 of 551 /score requests and 86 of 516 audit requests; the audit gates hold that sample to grading's tolerances.

   18 gates

   • ✓review prerequisites
   • ✓entry points and files
   • ✓train.sh · Given TRAIN_BUDGET_SECONDS=240, exited 0 on NVIDIA H100 80GB HBM3 after 228 s, writing 5 files (352489446 bytes) to OUTPUT_DIR. End of its log: [model 0] step 500 p=0.532 T=4096 B=16 lr=0.937 loss=2.7225 ema=4.6745 epochs=1.75 elapsed=77s [model 0] step 550 p=0.585 T=4096 B=16 lr=0.831 loss=2.5570 ema=4.4627 epochs=1.92 elapsed=85s [model 0] step 600 p=0.638 T=4096 B=16 lr=0.724 loss=2.7318 ema=4.2896 epochs=2.10 elapsed=92s [model 0] step 650 p=0.691 T=4096 B=16 lr=0.618 loss=2.7415 ema=4.1348 epochs=2.27 elapsed=100s [model 0] step 700 p=0.744 T=4096 B=16 lr=0.512 loss=2.4119 ema=3.9625 epochs=2.44 elapsed=108s [model 0] step 750 p=0.797 T=4096 B=16 lr=0.406 loss=2.6382 ema=3.8301 epochs=2.62 elapsed=115s [model 0] step 800 p=0.850 T=4096 B=16 lr=0.300 loss=2.5734 ema=3.7044 epochs=2.79 elapsed=123s [model 0] step 850 p=0.903 T=4096 B=16 lr=0.194 loss=2.3562 ema=3.5696 epochs=2.97 elapsed=131s [model 0] step 900 p=0.956 T=4096 B=16 lr=0.087 loss=2.2704 ema=3.4397 epochs=3.14 elapsed=138s [model 0] done steps=941 epochs=3.29 train_time=200s compile status: {'ns': True, 'model': True} total wall time since replay start: 227s
   • ✓train.sh wall clock · Grading's hard limit, with TRAIN_BUDGET_SECONDS=3600; this run had TRAIN_BUDGET_SECONDS=240.
   • ✓startup · GET /health answered 200 after 4.8 s
   • ✓GET /vocab · {"tokens": 2048, "special": 1}
   • ✓POST /score · {"id": "validation-0001", "bytes": 5203, "tokens": 1957}
   • ✓POST /score, characters outside the corpus · {"bytes": 1035, "tokens": 997}
   • ✓POST /dist · {"prefix_tokens": 978, "logprobs": 2048, "logsumexp": 2.220446049250313e-16, "score_minus_dist": 6.06516459278339e-08}
   • ✓evaluation deadline · Projected from 177 of 1067 requests at concurrency 8: 91 of 551 /score requests on public documents in 26.6 s and 86 of 516 /dist and truncation requests in 14.4 s, each scaled to its full count, plus the first requests (1.2 s) and the restart and its first /dist (9 s) as measured
   • ✓slowest request · 183 answered requests, median 1.65 s
   • ✓restart startup · GET /health after the kill and the wipe of candidate state
   • ✓requests answered · Every request was answered.
   • ✓/dist normalization · max |logsumexp| over 84 audited /dist positions
   • ✓/score vs /dist · max |score - dist| over the same positions
   • ✓/score vs /dist bias · mean(score - dist) over the same positions
   • ✓truncated text · max difference over 3479 shared positions of 2 truncated texts
   • ✓healthy after requests

The agent's last message before its session ended.