Speculative drafting

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.

1. 1
   failed7h 23m into the runtook 6m 4s

   failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ation_utils_base.py", line 1209, in _set_model_specific_special_tokens self.SPECIAL_TOKENS_ATT

   3 gates

   • ✓review prerequisites
   • ✓submission size
   • ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: ation_utils_base.py", line 1209, in _set_model_specific_special_tokens self.SPECIAL_TOKENS_ATTRIBUTES = self.SPECIAL_TOKENS_ATTRIBUTES + list(special_tokens.keys()) ^^^^^^^^^^^^^^^^^^^ AttributeError: 'list' object has no attribute 'keys' During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/submission/server.py", line 133, in <module> main() File "/submission/server.py", line 122, in main load() File "/submission/server.py", line 34, in load _tok = AutoTokenizer.from_pretrained(MODEL_DIR, trust_remote_code=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/models/auto/tokenization_auto.py", line 1140, in from_pretrained return tokenizer_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2097, in from_pretrained return cls._from_pretrained( ^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 2344, in _from_pretrained except import_protobuf_decode_error(): ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/venv/lib/python3.12/site-packages/transformers/tokenization_utils_base.py", line 87, in import_protobuf_decode_error raise ImportError(PROTOBUF_IMPORT_ERROR.format(error_message)) ImportError: requires the protobuf library but it was not found in your environment. Check out the instructions on the installation page of its repo: https:/ ...[clipped]
2. 2
   passed7h 33m into the runtook 5m 22s

   startup 72.4 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).

   7 gates

   • ✓review prerequisites
   • ✓submission size
   • ✓startup · GET /health answered 200 after 72.4 s
   • ✓POST /draft · {"draft": [8420, 374, 279, 821, 16099, 1119, 27445, 3561, 448, 279, 7102, 1172, 3055, 1447, 73594, 18104]}
   • ✓held-out serving time (projected) · projected from 584 of about 10,641 held-out requests at concurrency 32: 191.9 s of serving over 30 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.4 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
   • ✓request failures · 0 of 584 requests failed, timed out or were malformed; each scores as an empty draft
   • ✓healthy after requests
3. 3
   passed9h 43m into the runtook 4m 36s

   startup 74.3 s of 300 s; 2/2 gates passed; The serving gate was projected from a sample: 76-token windows, spread from openings to ends, of the first 32 of the 40 public conversations (2,432 of the held-out set's 44,314 continuation tokens), walked under the budget scaled to them (593 s).

   7 gates

   • ✓review prerequisites
   • ✓submission size
   • ✓startup · GET /health answered 200 after 74.3 s
   • ✓POST /draft · {"draft": [73594, 18104, 198, 2528, 11, 9669, 717, 3658, 8402, 18670, 4698, 1639, 11, 15, 62, 21]}
   • ✓held-out serving time (projected) · projected from 558 of about 10,167 held-out requests at concurrency 32: 148.5 s of serving over 31 rounds for 2,432 public continuation tokens, scaled to the held-out set's 44,314 as validate.py does, plus the smoke request's 0.3 s once; within about 10% of the held-out serving time: a few percent high for a server that answers requests one at a time, up to 10% low for one that batches the requests in flight
   • ✓request failures · 0 of 558 requests failed, timed out or were malformed; each scores as an empty draft
   • ✓healthy after requests

The agent's last message before its session ended.