Ternary quantization
- Score
- 5.0
- Wallet spend
- $31.65 of $200
- Model cost
- $2.33
- Agent time
- 9h 31m
- Steps
- 865
- Tokens in / out
- 88.2M / 771k
- Self-checks
- 2 passed of 3 used
- Rank at $200 wallet
- 10th of 19
Score
5.0raw metric 0.1147
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.217 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0340 | 0 | 0.866 | 3.9 |
| livecodebench_v6 | 0.00474 | 0 | 0.425 | 1.1 |
| mmlu_pro | 0.203 | 0.111 | 0.716 | 15.2 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0160
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned qwen startIn trajectory step 51 the base is downloaded with snapshot_download(repo_id="Qwen/Qwen3.5-4B", revision="851bf6e806efd8d
In trajectory step 51 the base is downloaded with snapshot_download(repo_id="Qwen/Qwen3.5-4B", revision="851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a"). That matches the pinned revision. GPTQ quantization (src/quantize.py, src/ternary.py) runs on that float model and STE QAT (src/qat.py) follows. The quantize manifest records the same base and revision. The trusted ternary audit finds every language-model tensor referenced, with nothing unreferenced. Only model.visual.* and mtp.* are omitted, so the checkpoint layout is the Qwen3.5-4B text tower. I found no other initialization.
✓text only inferenceserve.sh starts only src/server.py. The server builds Qwen3_5ForCausalLM, the text tower only, through build_text_model
serve.sh starts only src/server.py. The server builds Qwen3_5ForCausalLM, the text tower only, through build_text_model in src/model_utils.py. It cuts multimodal message content down to its text parts. No vision tower or MTP weights are shipped or loaded, and the trusted audit shows model.visual.* and mtp.* omitted.
✓no public benchmark eval trainingEvery corpus is a declared public train split, listed in architecture.json and evidence/data_manifest.json: C4 train, Ul
Every corpus is a declared public train split, listed in architecture.json and evidence/data_manifest.json: C4 train, UltraChat train_sft, OpenBookQA/ARC train, CodeFeedback, code_contests train, GSM8K train, and Hendrycks MATH train (src/prepare_math_data.py; evidence/prep_math.log reads split=train). The numeric MC set comes from GSM8K train, and the teacher traces come from MC train prompts. The data manifest says UltraChat's test splits are never read. The trajectory touches gpqa_diamond.csv and livecodebench only to list files and run /task/validate.py. No training or data-prep source reads them. Download logs are not retained for every corpus, so those sources count as declared but unverified.
✓actual inference uses reviewed weightsserve.sh runs src/server.py with --weights $HERE/weights and --manifest quantization_manifest.json. load_state_dict read
serve.sh runs src/server.py with --weights $HERE/weights and --manifest quantization_manifest.json. load_state_dict reads every tensor listed in the manifest from weights/model-0000{1,2,3}.safetensors. build_text_model raises an error if any weight is missing and ties lm_head to embed_tokens. No other model is loaded at serve time, and load_original_text_model is used only by make_traces.py. These are the same files the trusted audit covers, with 0 unreferenced parameters.
✓ternary executionFor packed_2bit tensors, server.py calls unpack_codes and then dequantize(codes, scale, group_size). ternary.py defines
For packed_2bit tensors, server.py calls unpack_codes and then dequantize(codes, scale, group_size). ternary.py defines dequantize as per-group fp16 scale times codes in {-1,0,+1}. The trusted ternary audit passed with a ternary fraction of 0.96944 and the parameter cap met. Only embed_tokens (tied) and small float tensors are left in float.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENROUTER, api_key, 'claude -p' and codex found no participant use of harness creden
Trajectory searches for ANTHROPIC, OPENROUTER, api_key, 'claude -p' and codex found no participant use of harness credentials or base URLs, and no launches of extra harness CLIs. Remote compute went through the allowed spin CLI, as in step 51.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 8m into the runtook 1m 22sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
- 2failed1h 12m into the runtook 6m 37s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] python: /opt/venv/bin/python3 (Python 3.12.3) [serve] installing the vendored wheels into
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] python: /opt/venv/bin/python3 (Python 3.12.3) [serve] installing the vendored wheels into /tmp/ternary_pkgs [serve] pip install finished [serve] starting the server on 0.0.0.0:8000 (model=submission) Error importing huggingface_hub.file_download: No module named 'httpx2' Traceback (most recent call last): File "/submission/src/server.py", line 30, in <module> from model_utils import build_text_model, read_text_config # noqa: E402 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/src/model_utils.py", line 13, in <module> from transformers.models.qwen3_5.configuration_qwen3_5 import Qwen3_5TextConfig File "/tmp/ternary_pkgs/transformers/__init__.py", line 30, in <module> from . import dependency_versions_check File "/tmp/ternary_pkgs/transformers/dependency_versions_check.py", line 16, in <module> from .utils.versions import require_version, require_version_core File "/tmp/ternary_pkgs/transformers/utils/__init__.py", line 77, in <module> from .hub import ( File "/tmp/ternary_pkgs/transformers/utils/hub.py", line 29, in <module> from huggingface_hub import ( File "/tmp/ternary_pkgs/huggingface_hub/__init__.py", line 1230, in __getattr__ submod = importlib.import_module(submod_path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/importlib/__init__.py", line 90, in import_module return _bootstrap._gcd_import(name[level:], package, level) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/ternary_pkgs/huggingface_hub/file_download.py", line 16, in <module> import httpx2 ModuleNotFoundError: No module named 'httpx2'
- 3passed6h 31m into the runtook 4m 45s
startup 83.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 83.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 1768, "content_end": "care.\n\nTherefore, the correct answer is that all operators must respect both Lorentz and Poincare symmetries.\n\nANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 101.7 s, median 101.7 s; 4 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed9h 21m into the runtook 6m 40s
startup 69.3 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 69.3 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 599, "content_end": "{3}{4} \\hbar^2 L^2$. Since there are four such states, the total energy of the system is $\\boxed{4\\cdot E}$.\n\nANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 212.3 s, median 212.2 s; 2 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 05:10:36 UTC.
What shipped
The model — Qwen3.5-4B text tower, ternary weights, 5 training phases:
| Ternary fraction of audited LM params | 0.9694 (required > 0.95) — 3,569,090,560 codes, one per original weight |
| Storage overhead | 1.0312× (limit 1.10×) — codes packed 2-bit + one fp16 scale per (row, 32 channels) |
| Exempt (float) | input embedding / tied head, RMSNorms, A_log/dt_bias/conv1d; vision tower and MTP head dropped (text-only) |
Pipeline: ternary GPTQ with an exact per-group scale search → QAT of the codes (straight-through estimator, EMA gradient, frozen scales) over five successive mixtures, each adding one answer format the previous phase was missing: chat → multiple choice → code → maths (GSM8K + Hendrycks MATH, train splits) → multiple choice with numerical options (so the letter is answered, not the value).
The server — serve.sh installs 57 vendored wheels offline into /tmp (69 s startup of the 300 s budget), loads the ternary shards, and serves the OpenAI chat API with batched greedy decoding, a 1.1 repetition penalty (breaks the digit/LaTeX repetition loops a ternary model falls into), and a 4096-token cap matching the evaluator's own budget.
One inference-time addition (src/answer_format.py): when the model answers a multiple-choice question with the value of an option instead of its letter, the server states the letter of the matching option — only when the question really has an option list, only when the model named no letter itself, and only when the value matches exactly one option. The same module recovers the value of a maths answer when the model ends with a letter or states the result in a bare sentence. The reasoning is never altered and the model still has to pick the right option.
Measured (hand-written probes, no benchmark data used for training or selection): letter given on 10/10 numeric-option questions (7/10 correct), a parsable value on 10/10 free-form maths questions, 7/10 on hand-written knowledge questions. The self-check passed all gates: ternary fraction, storage, startup 69.3 s, slowest of 16 concurrent requests 212.3 s against the 300 s limit, 0 timeouts.
Provenance for every corpus (train splits only), the download commands, the QAT logs, and the conventions are in evidence/; architecture.json lists all training and inference files.