LLM post-training

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1. 1
   failed9h 49m into the runtook 10m 8s

   failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: r__ await self._router._startup() File "/tmp/vendor-pkgs/fastapi/routing.py", line 6455, in

   3 gates

   • ✓review prerequisites
   • ✓checkpoint audit
   • ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: r__ await self._router._startup() File "/tmp/vendor-pkgs/fastapi/routing.py", line 6455, in _startup await handler() File "/submission/server.py", line 66, in startup engine = AsyncLLMEngine.from_engine_args(args) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/engine/async_llm_engine.py", line 574, in from_engine_args return async_engine_cls.from_vllm_config( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/utils/__init__.py", line 1589, in inner return fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/v1/engine/async_llm.py", line 212, in from_vllm_config return cls( ^^^^ File "/tmp/vendor-pkgs/vllm/v1/engine/async_llm.py", line 114, in __init__ self.tokenizer = init_tokenizer_from_configs( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/transformers_utils/tokenizer_group.py", line 123, in init_tokenizer_from_configs return TokenizerGroup( ^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/transformers_utils/tokenizer_group.py", line 28, in __init__ self.tokenizer = get_tokenizer(self.tokenizer_id, **tokenizer_config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/transformers_utils/tokenizer.py", line 256, in get_tokenizer tokenizer = get_cached_tokenizer(tokenizer) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/vllm/transformers_utils/tokenizer.py", line 99, in get_cached_tokenizer tokenizer.all_special_tokens_extended) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/vendor-pkgs/transformers/tokenization_utils_base.py", line 1306, in __getattr__ raise AttributeError(f"{self.__class__.__name__} ...[clipped]
2. 2
   passed10h 10m into the runtook 8m 26s

   startup 156.3 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.

   7 gates

   • ✓review prerequisites
   • ✓checkpoint audit
   • ✓startup · GET /health answered 200 after 156.3 s
   • ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 5029, "content_end": "between the planets? A. ~ 1.30 Comparing the choices, 'A. ~ 1.30' is the one consistent with this. So the answer is A.\n"}
   • ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 141.1 s, median 107.3 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
   • ✓concurrent requests answered · 32 well-formed replies
   • ✓healthy after requests
3. 3
   passed10h 28m into the runtook 9m 44s

   startup 184.2 s of 300 s; 1/1 gates passed; One wave of 32 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.

   7 gates

   • ✓review prerequisites
   • ✓checkpoint audit
   • ✓startup · GET /health answered 200 after 184.2 s
   • ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 28059, "content_end": "it contradicts the interatomic distance. However, following the formula as provided, the calculation yields **0 ...[clipped]
   • ✓slowest of 32 concurrent requests · 32 public requests at once (16 GPQA, 16 hard LiveCodeBench) as graded (max_tokens 16384, temperature 0.7, 250 s each): slowest 142.3 s, median 119.4 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 32 at a time, scores a request over 250 s or still failing as wrong with no limit on how many; held-out requests may run longer.
   • ✓concurrent requests answered · 32 well-formed replies
   • ✓healthy after requests

The agent's last message before its session ended.