Ternary quantization
- Score
- 4.3
- Wallet spend
- $7.43 of $10
- Model cost
- $1.25
- Agent time
- 4h 22m
- Steps
- 548
- Tokens in / out
- 54.1M / 482k
- Self-checks
- 2 passed of 3 used
- Rank at $10 wallet
- 7th of 19
Score
4.3raw metric 0.1093
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.288 | 0.250 | 0.571 | 11.8 |
| math_500 | 0.0200 | 0 | 0.866 | 2.3 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.129 | 0.111 | 0.716 | 3.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe trajectory shows snapshot_download of Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /dev
The trajectory shows snapshot_download of Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /dev/shm/model. The quantize script defaults to that same repo and revision, and logs/quantize.log shows GPTQ loading the model from /dev/shm/model. Scale training starts from the GPTQ codes and scales, with the codes frozen. architecture.json declares the same base and revision. The trusted audit found 0 unreferenced parameters.
✓text only inferencecode/qtern/server.py builds a text-only Qwen3_5ForCausalLM from Qwen3_5TextConfig and serves a local /v1/chat/completion
code/qtern/server.py builds a text-only Qwen3_5ForCausalLM from Qwen3_5TextConfig and serves a local /v1/chat/completions endpoint with greedy decoding. It makes no external model or API calls. architecture.json and the trusted audit both state that the vision and MTP weights are left out.
✓no public benchmark eval trainingprepare_mixed_data.py and logs/arc_data_provenance.json use only train splits: the ARC-Easy and ARC-Challenge train parq
prepare_mixed_data.py and logs/arc_data_provenance.json use only train splits: the ARC-Easy and ARC-Challenge train parquets, GSM8K main train and the self-oss-instruct train split. The C4 train chunks are also declared. Format selection used 200 held-out ARC train questions. GPQA files appear only in eval and prompt-format probe runs, and their output feeds no weight update. The LiveCodeBench format suffix is copied as a prompt template only, with no benchmark examples. The download commands are retained, so provenance is declared.
✓actual inference uses reviewed weightsserve.sh runs code/launch.py, which calls qtern.server.main. That function reads quantization_manifest.json and loads we
serve.sh runs code/launch.py, which calls qtern.server.main. That function reads quantization_manifest.json and loads weights/ternary-codes.safetensors, weights/ternary-scales.safetensors and weights/float-tensors.safetensors from the submission directory. It loads them strictly, raising on missing or unexpected tensors, and ties lm_head to the embedding. These are exactly the files the trusted audit inspected.
✓ternary executionThe trusted ternary audit passed with a ternary fraction of 0.983 and 0 unreferenced parameters, and the parameter cap p
The trusted ternary audit passed with a ternary fraction of 0.983 and 0 unreferenced parameters, and the parameter cap passed. The server dequantizes the packed 2-bit {-1,0,+1} codes as code times a per-group float scale (pack.py) and keeps only the embedding, norms and small linear-attention tensors in float.
✓harness credentials used only by the harnessSearching the trajectory found no use, printing or copying of the harness API key or base URL, and no extra harness CLI
Searching the trajectory found no use, printing or copying of the harness API key or base URL, and no extra harness CLI launch (no claude -p or anthropic usage). The API_KEY and BASE_URL hits are Spin CLI help text and an argparse option name in validate.py.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed2h 19m into the runtook 6m 59s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/code/launch.py", line 25, in <module> m
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/code/launch.py", line 25, in <module> main() File "/submission/code/launch.py", line 21, in main server_main() File "/submission/code/qtern/server.py", line 433, in main engine.load() File "/submission/code/qtern/server.py", line 76, in load from transformers import AutoTokenizer File "/tmp/qtern-venv/transformers/__init__.py", line 30, in <module> from . import dependency_versions_check File "/tmp/qtern-venv/transformers/dependency_versions_check.py", line 56, in <module> require_version_core(deps[pkg]) File "/tmp/qtern-venv/transformers/utils/versions.py", line 116, in require_version_core return require_version(requirement, hint) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/qtern-venv/transformers/utils/versions.py", line 110, in require_version _compare_versions(op, got_ver, want_ver, requirement, pkg, hint) File "/tmp/qtern-venv/transformers/utils/versions.py", line 43, in _compare_versions raise ImportError( ImportError: safetensors>=0.8.0 is required for a normal functioning of this module, but found safetensors==0.5.3. Try: `pip install transformers -U` or `pip install -e '.[dev]'` if you're working with git main
- 2passed2h 28m into the runtook 4m 1s
startup 47.0 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 47.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 2546, "content_end": "y the human experience.\nI have a passion for the human experience.\nI am writing that has a passion for human experience."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 59.2 s, median 9.8 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed4h 14m into the runtook 4m 9s
startup 58.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 58.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 9, "content_end": "ANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 20.0 s, median 2.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-21 00:00:12 UTC.
Final submission
/workspace/output/ — 116 files, 6.38 GB
Quantization
| property | value | requirement |
|---|---|---|
| ternary share of text-model params (excl. embedding/head) | 98.33 % | > 95 % |
| stored params / original params | 1.0156× | ≤ 1.10× |
| ternary code granularity | one code per original weight, 2-bit packed (4/byte) | — |
| float parts kept | embedding, output head, activations, vision tower, MTP head | allowed |
Pipeline: GPTQ ternary pass → end-to-end training of the group scales only (custom autograd, exact scale × code gradient) → answer-format training (ternary codes frozen, scales only) → serving with dequantized ternary weights only.
Measured results
- Public GPQA-Diamond: 0.30 (limit 20) / 0.267 (limit 60), random guessing = 0.25
- Held-out ARC train letter accuracy: 0.844 (0.250 before the format stage)
- C4 dev perplexity: 25.84 (fp16 original: 14.4)
- 16 concurrent requests: 6.2 s on the A100 box / 20 s slowest on the H100, every answer
finish_reason: stop - Server startup: 58.2 s of the 300 s budget
frontier selfcheck(cc714ab239e1): PASSED, all gates green
Compliance
No weight was trained, calibrated, or selected on any validation/test split. Training data is C4 (train) + ARC/GSM8K/self-oss-instruct (train splits only) with hand-written instruction paraphrases; checkpoints were selected on data held out of their own training set (disjoint C4 slice, 200 held-out ARC train questions). The public benchmark examples were used only for measurement and for a five-way serving-rendering comparison that touched no weight — both disclosed in architecture.json and logs/prompt_format.md. The server never post-processes model output; the answer formats are learned by the model.
Evidence is in architecture.json (base model + revision, 426-tensor manifest mapping, corpora with split and download command, procedure, measurements, 35 training + 18 inference evidence files, none missing) and logs/ (training consoles, perplexity, GPQA reports, concurrency, startup, selfcheck report, weight audit).