Ternary quantization

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.

1. 1
   failed16m 11s into the runtook 6m 33s

   startup 95.8 s of 300 s; failed GPQA request as graded: the response was rejected: ValueError: response lacks exactly one choice; got {"error": {"message": "'tokenizers.Encoding' object has no attribute 'numel'", "type": "invalid_request_error"}}

   6 gates

   • ✓review prerequisites
   • ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
   • ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
   • ✓weight audit · inspect_ternary passed, as verify requires before scoring
   • ✓startup · GET /health answered 200 after 95.8 s
   • ✗GPQA request as graded · the response was rejected: ValueError: response lacks exactly one choice; got {"error": {"message": "'tokenizers.Encoding' object has no attribute 'numel'", "type": "invalid_request_error"}}
2. 2
   passed23m 10s into the runtook 13m 5s

   startup 90.4 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones a

   9 gates

   • ✓review prerequisites
   • ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
   • ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
   • ✓weight audit · inspect_ternary passed, as verify requires before scoring
   • ✓startup · GET /health answered 200 after 90.4 s
   • ✓GPQA request as graded · {"note": "no reply within 300 s: grading scores a timed-out request as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission HTTP request failed: TimeoutError: timed out"}
   • ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 300.2 s, median 300.2 s; 0 stopped at max_tokens, 16 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
   • ✓concurrent requests answered · 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.
   • ✓healthy after requests
3. 3
   passed37m 24s into the runtook 12m 9s

   startup 80.9 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 1 well-formed replies; 15 timed out after 300 s. Grading scores each such request as wrong, so 15 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones a

   9 gates

   • ✓review prerequisites
   • ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
   • ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
   • ✓weight audit · inspect_ternary passed, as verify requires before scoring
   • ✓startup · GET /health answered 200 after 80.9 s
   • ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1071, "content_end": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n ...[clipped]
   • ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 301.2 s, median 300.2 s; 0 stopped at max_tokens, 15 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
   • ✓concurrent requests answered · 1 well-formed replies; 15 timed out after 300 s. Grading scores each such request as wrong, so 15 of 16 here would score zero.
   • ✓healthy after requests
4. 4
   passed51m 9s into the runtook 9m 4s

   startup 94.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.

   9 gates

   • ✓review prerequisites
   • ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
   • ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
   • ✓weight audit · inspect_ternary passed, as verify requires before scoring
   • ✓startup · GET /health answered 200 after 94.6 s
   • ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 676, "content_end": "\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\n\ ...[clipped]
   • ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 181.7 s, median 181.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
   • ✓concurrent requests answered · 16 well-formed replies
   • ✓healthy after requests

The agent's last message before its session ended.