Ternary quantization
- Score
- 8.3
- Wallet spend
- $34.52 of $200
- Model cost
- $15.71
- Agent time
- 5h 18m
- Steps
- 481
- Tokens in / out
- 45.8M / 272k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 7th of 19
Score
8.3raw metric 0.09415
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0152 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.214 | 0 | 0.866 | 24.7 |
| livecodebench_v6 | 0.0332 | 0 | 0.425 | 7.8 |
| mmlu_pro | 0.114 | 0.111 | 0.716 | 0.5 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0365
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe session downloaded the pinned base: trajectory#25 runs `hf download Qwen/Qwen3.5-4B --revision 851bf6e806efd8d0a36b0
The session downloaded the pinned base: trajectory#25 runs `hf download Qwen/Qwen3.5-4B --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`. A permissions error forced a retry under /ephemeral/work, and evidence/logs/model_download.log matches that location. source/convert_text.py pins the same BASE and REVISION and extracts the model.language_model.* tensors. Every later stage starts from that converted base: GPTQ, float selection, scale tuning, QAT and finalize_weights.py (which reads source/model.safetensors and the GPTQ outputs). The manifest records the pinned base_revision. The trusted audit finds every language tensor referenced, with only the vision and MTP tensors omitted.
✓text only inferenceconfig.json declares architectures Qwen3_5ForCausalLM with model_type qwen3_5_text and no vision config. serve.py regist
config.json declares architectures Qwen3_5ForCausalLM with model_type qwen3_5_text and no vision config. serve.py registers the text-only Qwen35TextForCausalLM class from qwen35_text.py, which rejects multimodal features. No vision or MTP tensors were shipped, and the trusted audit records them as omitted. serve.sh serves only the OpenAI-compatible text endpoint.
✓no public benchmark eval trainingsource/prepare_calibration.py streams only the train splits of four public datasets: HuggingFaceFW/fineweb-edu sample-10
source/prepare_calibration.py streams only the train splits of four public datasets: HuggingFaceFW/fineweb-edu sample-10BT, HuggingFaceH4/no_robots, nvidia/OpenMathInstruct-2 and codeparrot/apps. The internal dev set is a held-out tail of those same streams. evidence/data_manifest_large.json records the resolved revisions and token hashes, and the prepare_data log keeps the download run. The public benchmarks (GPQA Diamond and LiveCodeBench v6) were used only by validate.py after the weights were frozen (trajectory#402). Searches of the trajectory and source found no benchmark data used for training, calibration or selection.
✓actual inference uses reviewed weightsThe serving path is serve.sh → serve.py → ternary_loader.TernaryModelLoader, registered as load format 'ternary'. The lo
The serving path is serve.sh → serve.py → ternary_loader.TernaryModelLoader, registered as load format 'ternary'. The loader reads only the entries listed in quantization_manifest.json, which point to weights/float.safetensors and weights/layer_000..031.safetensors. That is exactly the storage the trusted audit covered, and the audit found 0 unreferenced parameters. No uninspected files exist. All 180 wheels installed offline, including vllm-0.19.1, have approved upstream digests.
✓ternary executionternary_loader.py unpacks each uint8 packed_2bit tensor into codes (&3)-1 in {-1,0,+1}, multiplies by the FP32 scale for
ternary_loader.py unpacks each uint8 packed_2bit tensor into codes (&3)-1 in {-1,0,+1}, multiplies by the FP32 scale for each group of 32 columns, and passes through only the entries the manifest declares as float. The trusted audit passed with a ternary fraction of 0.951. Stored parameters are 3,679,399,936, under the 3,927,057,305.6 limit. The only excluded float tensors are lm_head and embed_tokens.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENROUTER, api_key, BASE_URL, claude -p, codex, import anthropic and import openai f
Trajectory searches for ANTHROPIC, OPENROUTER, api_key, BASE_URL, claude -p, codex, import anthropic and import openai found no use of harness credentials by the participant. HF_TOKEN was only tested for presence (`printenv HF_TOKEN >/dev/null`). The broker/judge text that appears is the task's validate.py source being read, and validation ran without judge credentials (trajectory#402).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed4h 59m into the runtook 11m
startup 255.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,499,130,880 of 3,679,399,936 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951006, which must exceed 0.95
- ✓stored parameters · 3,679,399,936 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 255.7 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 153.7 s, median 153.4 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed5h 12m into the runtook 9m 48s
startup 218.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,499,130,880 of 3,679,399,936 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951006, which must exceed 0.95
- ✓stored parameters · 3,679,399,936 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 218.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 129.4 s, median 129.4 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The second selfcheck passed with startup reduced to 218.5 seconds (81.5 seconds of margin). The slowest of 16 concurrent requests was 129.4 seconds, with zero timeouts, and all static audits passed. I’m verifying that the local directory is still byte-for-byte the checked package, then proceeding with the two-step final submission.