Ternary quantization
- Score
- 8.7
- Wallet spend
- $8.73 of $10
- Model cost
- $4.21
- Agent time
- 2h 43m
- Steps
- 199
- Tokens in / out
- 64.3M / 355k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 3rd of 14
Score
8.7raw metric 0.09674
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.00505 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.216 | 0 | 0.866 | 24.9 |
| livecodebench_v6 | 0.00948 | 0 | 0.425 | 2.2 |
| mmlu_pro | 0.156 | 0.111 | 0.716 | 7.5 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0127
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe bootstrap step downloaded Qwen/Qwen3.5-4B with snapshot_download pinned to revision 851bf6e806efd8d0a36b00ddf55e13cc
The bootstrap step downloaded Qwen/Qwen3.5-4B with snapshot_download pinned to revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. qat.py pins the same revision (REV) and loads model.language_model.* from that local checkpoint with a strict check. Initialization is GPTQ-style ternary, followed by KL distillation against the bf16 teacher. export.py builds the ternary, float and vision tensors from the original checkpoint plus the trained state. The manifest records the pinned base and revision, and the final manifest in the trajectory shows revision 851bf6e8 with 426 tensors.
✓text only inferenceserve.sh launches the vLLM OpenAI server with --language-model-only. build_serving_checkpoint.py only decodes manifest t
serve.sh launches the vLLM OpenAI server with --language-model-only. build_serving_checkpoint.py only decodes manifest tensors and copies the vision tower unchanged. vision.safetensors holds only model.visual.* keys, which this flag leaves unused. No MTP head is shipped, and the embedding is tied to the output head.
✓no public benchmark eval trainingTraining and calibration data come only from declared training splits: fineweb-edu sample-10BT train, NuminaMath-CoT tra
Training and calibration data come only from declared training splits: fineweb-edu sample-10BT train, NuminaMath-CoT train, OpenCodeReasoning split_0 and smoltalk train. These are listed in evidence/data/manifest.json and streamed by prep_data.py. GPTQ calibration uses train.npy[:128], and QAT trains on train.npy with a holdout of every 50th document. GPQA-diamond was copied to the host only for a serve/validation smoke test and appears in no training or calibration path. The public sources have no retained download log, so their provenance is declared but unverified.
✓actual inference uses reviewed weightsserve.sh runs build_serving_checkpoint.py on the shipped submission. That script reads quantization_manifest.json, decod
serve.sh runs build_serving_checkpoint.py on the shipped submission. That script reads quantization_manifest.json, decodes weights/ternary.safetensors, copies weights/float.safetensors and weights/vision.safetensors, and vLLM then serves that built directory. No other weight files exist, and the hub is offline (HF_HUB_OFFLINE=1). The trusted audit passed with 0 unreferenced parameters. The only uninspected files are Triton launcher and cuda_utils .so cache objects, not model weights.
✓ternary executionAll mlp, self_attn and linear_attn projection linears are stored as packed 2-bit codes {-1,0,+1} with fp32 scales per gr
All mlp, self_attn and linear_attn projection linears are stored as packed 2-bit codes {-1,0,+1} with fp32 scales per group of 64. They are decoded at startup as code*scale; any unknown encoding raises an error. float.safetensors holds no float copies of those projections, only norms, conv, A_log/dt_bias, the small in_proj_a/b tensors and the tied embedding. The trusted audit reports a ternary fraction of 0.983, with only the tied embedding excluded.
✓harness credentials used only by the harnessNo trajectory step uses harness or judge credentials. MLE_JUDGE_API_KEY appears only in the validator's argument listing
No trajectory step uses harness or judge credentials. MLE_JUDGE_API_KEY appears only in the validator's argument listing. The validator package was uploaded for a local smoke test, with no sign of a judge key being used. HF_TOKEN is the participant's provided read-only download token. The curl calls go only to package indexes and localhost health checks.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 7m into the runtook 6m 22s
startup 158.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 158.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.5 s, median 23.5 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed2h 44m into the runtook 6m 6s
startup 156.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 156.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.8 s, median 23.8 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and graded as it stands. It passed both grader self-checks, but I expect weak scores on the reasoning benchmarks: most greedy answers are cut off at the 4096-token cap.
What was submitted (/workspace/output, 8.6 GB)
- Weights: ternary codes {-1, 0, +1} for all 200 language-model linears (MLP, attention, and linear-attention projections), with one fp32 scale per 64 columns. Norms, conv1d, the gate projections, and the tied embedding stay floating point. Vision tower is bf16 as-is; MTP is not shipped.
- Ratios from my audit and the grader's report: ternary is 98.3% of body parameters with scales counted as non-ternary (required > 95%). Stored parameters are 1.016× the original (limit 1.10×).
- Method: GPTQ-style ternary init, then two phases of distillation QAT from the original bf16 model. Held-out KL went from 2.00 at init to 0.366 after phase 1 and 0.356 after phase 2. Top-1 agreement with the teacher is 79%.
- Serving:
serve.shinstalls the 179 vendored cu128 wheels into/tmp, decodes the packed codes to bf16, and launches vLLM 0.19.1 with the triton GDN backend. Prewarmed Triton and Inductor caches ship incache/. - Evidence:
architecture.json, training source, logs, the data manifest (training splits only, held-out documents never trained on), and an audit script.
Verification
- Two grader self-checks passed. Startup took 156.5 s of 300 s. All 16 concurrent public requests returned valid responses.
- A cold start from an empty
/tmpreached/health200 in 160 s on my own H100 run.
Main risk: truncation
- In the self-check, 15 of 16 public GPQA/LCB requests stopped at the 4096-token cap, which grading scores as zero. The traces are long and often second-guess themselves ("Wait, I made a mistake…"), and some repeat.
- I tested two inference-side fixes on training-split math problems: a repetition penalty and disabling thinking. Neither reduced truncation, so I kept the model's default thinking mode.
- My boxed-answer accuracy on that proxy was wrong (the model writes bold answers, not
\boxed{}), so I only used the truncation numbers. - I did not measure the original model under this harness, so I can't say how far below the anchor this lands.
Decisions for you
- If the anchor was measured with thinking on and most originals also finish within 4096, the truncation problem is a real loss. Switching the default to non-thinking (
enable_thinking=false) is the main lever left; I did not make that change.
Budget and honesty notes
- Spent $8.73 of the $10 wallet on H100 leases.
- I ran a 16-item GPQA smoke test on the host to confirm the endpoint and output format. I did not use its results to choose any setting.