Ternary quantization
- Score
- 0.0
- Wallet spend
- $4.50 of $10
- Model cost
- $36.36
- Agent time
- 10h 16m
- Steps
- 474
- Tokens in / out
- 101.1M / 387k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 11th of 14
Score
0.0raw metric 0.007754
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0303 | 0.250 | 0.571 | 0.0 |
| math_500 | 0 | 0 | 0.866 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.000714 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startevidence/quantize3.py runs post-training GPTQ on the base model at /root/model, reading the original model*.safetensors.
evidence/quantize3.py runs post-training GPTQ on the base model at /root/model, reading the original model*.safetensors. That base was downloaded with snapshot_download at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (evidence/data_manifest.json); the task's STARTING_ARTIFACT also names this revision (trajectory#3). Later scale and distillation tuning keeps the ternary codes frozen and only adjusts scales and float tensors, so the weights come from the pinned Qwen3.5-4B. quantization_manifest.json records the same base revision.
✓text only inferenceserver.py builds a Qwen3_5ForCausalLM from Qwen3_5TextConfig (model_config.json) and serves only /v1/chat/completions wi
server.py builds a Qwen3_5ForCausalLM from Qwen3_5TextConfig (model_config.json) and serves only /v1/chat/completions with text input. The vision tower (model.visual.*) and MTP head are omitted from weights.safetensors. The trusted ternary audit lists them as omitted parameters.
✓no public benchmark eval trainingThe training and calibration corpora are declared in evidence/data_manifest.json. All are train splits of public dataset
The training and calibration corpora are declared in evidence/data_manifest.json. All are train splits of public datasets: wikitext-103, codeparrot-clean, WebInstructSub, MetaMathQA, sciq, ai2_arc, openbookqa and qasc. Download commands are declared, but full download logs are only partly verifiable. data_prep*.py loads train splits only. data_prep5.py copies the GPQA answer-suffix wording only, with no benchmark rows. In the trajectory, /task/public/benchmarks was only listed and its CSV header inspected (trajectory#3, trajectory#355). Behavioural checks used hand-written prompts (trajectory#352, trajectory#361). No benchmark rows were used for training or selection.
✓actual inference uses reviewed weightsserve.sh installs the bundled wheels and runs server.py. server.py reads quantization_manifest.json and loads every tens
serve.sh installs the bundled wheels and runs server.py. server.py reads quantization_manifest.json and loads every tensor from weights.safetensors: packed codes are dequantized and float tensors are loaded as stored. It then calls load_state_dict with an assertion that there are no unexpected keys. No other checkpoint or model is loaded. The trusted audit's audited storage is this same weights.safetensors, with zero unreferenced parameters.
✓ternary executionThe layer projections are stored packed_2bit as codes in {-1,0,+1} with per-group float scales. server.py serves them as
The layer projections are stored packed_2bit as codes in {-1,0,+1} with per-group float scales. server.py serves them as (code-1)*scale; nothing else is substituted. Float tensors are limited to the embeddings, the untied lm_head, norms, conv1d, A_log/dt_bias and the small in_proj_a/b. The trusted ternary audit passed, with a ternary fraction of 0.9684 and the parameter cap met.
✓harness credentials used only by the harnessThe trajectory shows compute used only through the Spin CLI (spin compute exec/upload/list) and HF_TOKEN used only for d
The trajectory shows compute used only through the Spin CLI (spin compute exec/upload/list) and HF_TOKEN used only for downloads. There were no direct API-key or base-URL calls and no launches of other harness CLIs. OpenRouter appears only in harness metadata and Spin help text (trajectory#1, trajectory#358).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed4h 11m into the runtook 16m 43s
startup 158.3 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 158.3 s
- ✓GPQA request as graded · {"note": "no reply within 300 s: grading scores a timed-out request as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission returned HTTP 500: b'{\"error\": {\"message\": \"generation timeout\"}}'"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 291.4 s, median 290.4 s; 0 stopped at max_tokens, 16 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 2passed6h 25m into the runtook 7m 49s
startup 182.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 182.7 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 103.2 s, median 56.9 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed9h 38m into the runtook 8m 4s
startup 168.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 168.1 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 102.1 s, median 55.4 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed10h 8m into the runtook 8m 5s
startup 164.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 164.1 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 109.6 s, median 55.5 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-13 04:54 UTC. The final submission in /workspace/output/ passed frontier selfcheck (run f9da407dd832) with all gates green.
What was shipped:
- Quantization: sequential ternary GPTQ (group-32 per-row scales, packed 2-bit codes, staged Hessian capture) on Qwen3.5-4B, keeping embedding/head, vision tower, MTP, norms and small float tensors in floating point. Audit: 96.84% ternary (>95% required), stored non-exempt params 3.681B vs 3.927B limit.
- Quality recovery: scale/float tuning plus four rounds of KL distillation against the frozen FP teacher, all on training-split corpora only (wikitext-103 train, codeparrot, WebInstructSub, MetaMathQA). Internal held-out-train PPL went 85.5 → 38.9.
- Round-4 format distillation (the decisive change): starting from round 3, trained on SciQ/ARC/OpenBookQA/QASC train splits rendered in the evaluator's multiple-choice format with brief reasoning ending in
ANSWER: X+<|im_end|>, mixed with math/instruct/code data. On hand-written probe questions (no benchmark data used for training or selection), this moved stopping from 0/12 (round 2) and answer-marker compliance from 0/12 (round 3) to stop 8/12, marker 5/12, correct 3/12. - Server: offline OpenAI-compatible endpoint via vendored wheels, custom decode loop with CUDA graph; selfcheck measured startup 164.1s (limit 300s) and slowest of 16 concurrent graded-style requests 109.6s (limit 300s).
Known limitation: the ternary model is still heavily degraded versus the FP base (internal PPL ~39 vs 10.9), and on hard GPQA-style prompts many replies still run to max_tokens and score zero; the submission earns credit where generations stop with a parseable answer. Evidence (quantizer, data-prep and distillation sources, logs, data manifest with provenance) is included under /workspace/output/evidence/.