Ternary quantization
- Score
- 18.9
- Wallet spend
- $35.89 of $50
- Model cost
- $42.04
- Agent time
- 10h 34m
- Steps
- 457
- Tokens in / out
- 45.6M / 420k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 3rd of 14
Score
18.9raw metric 0.2115
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.283 | 0.250 | 0.571 | 10.2 |
| math_500 | 0.288 | 0 | 0.866 | 33.3 |
| livecodebench_v6 | 0.0730 | 0 | 0.425 | 17.2 |
| mmlu_pro | 0.202 | 0.111 | 0.716 | 15.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1559
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a with snapshot
The participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a with snapshot_download into /home/Ubuntu/models/qwen3.5-4b (trajectory step 60). The start artifact's metadata confirms the same repo and revision (trajectory steps 94 and 210). Every pipeline script uses that path. pipeline/export_ternary.py builds the starting checkpoint by MSE-optimal ternary quantization of the original weights, logged in logs/export_init.log. QAT training starts from that checkpoint, and the bf16 teacher is the same unmodified model. architecture.json declares the same base model and revision.
✓text only inferenceserve.sh starts vLLM with --language-model-only on the materialized text model. decode_guard.py only forwards text promp
serve.sh starts vLLM with --language-model-only on the materialized text model. decode_guard.py only forwards text prompts to the local /v1/completions endpoint. The trusted ternary audit confirms that the vision tower tensors are left out of the shipped weights.
✓no public benchmark eval trainingdata_manifest.json and pipeline/build_prompts.py declare public train splits only, all downloaded with hf_hub_download:
data_manifest.json and pipeline/build_prompts.py declare public train splits only, all downloaded with hf_hub_download: gsm8k, MATH, NuminaMath, OpenMathInstruct-2, MMLU aux_train, sciq, ARC, medmcqa, APPS, TACO, CodeFeedback, Magicoder, ultrachat, tulu-3, smollm and open-web-math. None of these is the GPQA or LiveCodeBench eval data. Only the build log exists, so the download of each source is unverified, but every source is declared and public. Checkpoint selection used held-out KL on the training distribution (trajectory step 322). Guard and MCQ tuning used dev sets drawn from held-out train-split prompts (trajectory steps 431 and 434). The validate.py runs on gpqa_diamond and LCB only measured and diagnosed the model. The LCB prompt-format and regex checks (trajectory step 448) tested format compatibility and were not training or selection.
✓actual inference uses reviewed weightsserve.sh runs materialize.py on the submission's weights/ shards. materialize.py rebuilds bf16 weights as code × group s
serve.sh runs materialize.py on the submission's weights/ shards. materialize.py rebuilds bf16 weights as code × group scale and copies the float tensors unchanged. serve.sh then serves only that materialized directory with vLLM. decode_guard.py sends requests only to the local vLLM backend at 127.0.0.1:8001 and loads no other model or external API.
✓ternary executionThe trusted ternary audit passed. It found a ternary fraction of 0.96841, 3.68e9 stored parameters (under the limit) and
The trusted ternary audit passed. It found a ternary fraction of 0.96841, 3.68e9 stored parameters (under the limit) and no unreferenced parameters. The projection weights are stored as packed_2bit codes {-1,0,+1} with a bf16 scale per group of 32. materialize.py decodes them faithfully to code × scale, so the served weights are exactly the ternary values.
✓harness credentials used only by the harnessThe trajectory shows no launches of the claude or codex CLI, and no harness API keys or base URL being used, copied or e
The trajectory shows no launches of the claude or codex CLI, and no harness API keys or base URL being used, copied or echoed. The `spin transfer credentials` and spin s5cmd calls (trajectory steps 60 and 307) belong to the allowed spin tooling. The model download used the allowed HF access.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 49m into the runtook 5m 20s
startup 131.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 131.6 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 25.2 s, median 24.7 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed10h 23m into the runtook 5m 56s
startup 109.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 109.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 5027, "content_end": "ides the correct product (hydroxy ketone).\n\nTherefore, the correct pair is Starting Material C and Product D.\n\nANSWER: D"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 73.5 s, median 60.9 s; 5 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output was handed in at 10:19:49 UTC, and the session is closed.
The last self-check passed on the same files: 96.8% of audited parameters are ternary, stored parameters are 1.03× the original count, startup took 110 s, and all 16 concurrent requests got well-formed replies.
On the unsubmitted-file side, the only change after that check was to the QUANTIZATION.md text, which now describes the final decode_guard.py (6-sample voting for multiple-choice and math, sample-test selection for code). serve.sh is unchanged.
Quality is the main risk:
- GPQA-Diamond: the guarded server scored 8 of 48 (16.7%) on the first 48 questions, which is below chance. The final score is a harmonic mean of the four benchmarks, each normalized from random guessing (25% on GPQA) to the original model. If GPQA lands at or below 25%, the total is zero.
- LiveCodeBench: the guarded server scored 2 of 48 (4%) on the first 48 problems. In the self-check, 5 of the 16 requests (the 8 LiveCodeBench ones were among them) hit the 4,096-token cap and would score zero.
- Direct answer scoring: I tried scoring answer letters directly with option shuffling on 200 train-split multiple-choice questions. It did not beat the sampled-reasoning vote (54% vs 61%), so I left it out.
The compute box is stopped, with about $35.9 of the $50 budget spent.