Ternary quantization
- Score
- 36.5
- Wallet spend
- $46.06 of $50
- Model cost
- $93.36
- Agent time
- 9h 57m
- Steps
- 284
- Tokens in / out
- 73.1M / 237k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 1st of 14
Score
36.5raw metric 0.311
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.136 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.620 | 0 | 0.866 | 71.6 |
| livecodebench_v6 | 0.173 | 0 | 0.425 | 40.8 |
| mmlu_pro | 0.314 | 0.111 | 0.716 | 33.6 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2236
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startIn trajectory#29 the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8c
In trajectory#29 the participant downloaded Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. QAT (quantization-aware training with KL distillation) started from that checkpoint, and export.py packed the shipped ternary codes from the trained latent weights. architecture.json and quantization_manifest.json declare the same base model and revision. Shipped tensor names and shapes match the Qwen3.5-4B structure.
✓text only inferenceserve.sh starts vLLM with limit-mm-per-prompt image=0 and video=0, and with no speculative or MTP decoding. code/front.p
serve.sh starts vLLM with limit-mm-per-prompt image=0 and video=0, and with no speculative or MTP decoding. code/front.py builds only text chat prompts (token ids sent to /v1/completions). The vision and MTP tensors are loaded into the checkpoint only so the architecture can be built; the README says they are not used for inference, and the serving settings disable them.
✓no public benchmark eval trainingcode/training/build_prompts.py and evidence/data_manifest.json declare training-split sources only: APPS, CodeContests t
code/training/build_prompts.py and evidence/data_manifest.json declare training-split sources only: APPS, CodeContests train, SciQ, ARC, OBQA, MedMCQA, GSM8K train, hendrycks_math train and ultrachat train_sft. Dev holdouts are carved from those train splits. Calibration in sens.py uses teacher generations on the same prompts. The trajectory shows public benchmark data (GPQA diamond, trajectory#219 and trajectory#265) used only for a post-training sanity evaluation of the final model, never for training or calibration. The downloads were logged by the scripts but are partly unverified from retained logs.
✓actual inference uses reviewed weightsserve.sh runs code/prepare_model.py, which reads quantization_manifest.json. It dequantizes only the entries in weights/
serve.sh runs code/prepare_model.py, which reads quantization_manifest.json. It dequantizes only the entries in weights/model.safetensors, which is the audited storage, and adds only model.visual.* and mtp.* keys from weights/extra_vision_mtp.safetensors (no language-model or lm_head keys). It then serves that /tmp/tq-model with vLLM through front.py, so no other weights reach serving.
✓ternary executionLinear weights are stored as packed_2bit codes {-1, 0, +1} with fp16 per-128 group scales. prepare_model.py reconstructs
Linear weights are stored as packed_2bit codes {-1, 0, +1} with fp16 per-128 group scales. prepare_model.py reconstructs each weight exactly as code × scale, and unknown encodings raise an error. The trusted ternary audit passed: ternary fraction 0.9547 (more than 0.95), 0 unreferenced parameters, and the parameter cap passed.
✓harness credentials used only by the harnessSearches of the trajectory found no use of OPENROUTER, api_key, harness BASE_URL, claude -p or codex. HF_TOKEN was used
Searches of the trajectory found no use of OPENROUTER, api_key, harness BASE_URL, claude -p or codex. HF_TOKEN was used only for read downloads (snapshot_download in trajectory#29, a dataset hf_hub_download); there were no HF uploads. The 'upload' matches are spin compute or scp copies to the participant's own GPU boxes. The teacher model ran on local vLLM through code/training/gen_client.py.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed2h 16m into the runtook 5m 32s
startup 122.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,434,086,400 of 3,596,880,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.954740, which must exceed 0.95
- ✓stored parameters · 3,596,880,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 122.4 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.3 s, median 18.2 s; 6 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed9h 38m into the runtook 5m 6s
startup 127.8 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,434,086,400 of 3,596,880,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.954740, which must exceed 0.95
- ✓stored parameters · 3,596,880,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 127.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 9650, "content_end": "e is achiral.\n\n Conclusion:\n A = Benzoquinone\n B = Methyl 2-hydroxypropanoate\n\n Matches Option D.\n\nANSWER: D"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.3 s, median 27.2 s; 11 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed9h 53m into the runtook 4m 26s
startup 95.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,434,086,400 of 3,596,880,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.954740, which must exceed 0.95
- ✓stored parameters · 3,596,880,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 95.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 9876, "content_end": "\n\nTherefore, the intended answer is the 8-carbon acid, which is named \"3,4-dimethyl-5,6-dioxo-octanoic acid\".\n\nANSWER: D"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.3 s, median 26.5 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
I'll remove the GPQA validation stdout from the evidence (it contains model outputs on benchmark questions; the README already records that one sanity run happened):