Ternary quantization
- Score
- 0.0
- Wallet spend
- $9.89 of $10
- Model cost
- $1.69
- Agent time
- 2h 49m
- Steps
- 177
- Tokens in / out
- 14.4M / 71k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 15th of 19
Score
0.0raw metric 0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0 | 0 | 0.866 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trajectory confirms
architecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trajectory confirms the download: snapshot_download fetched that repo at that exact revision into /workspace/qwen3.5-4b. quantize.py loads that directory directly in bf16 and applies data-free absmean PTQ. trajectory#80 is the quantize run that produced quant_final, and trajectory#121 copies its weights and manifest into /workspace/output. trajectory#172 lists the output files: weights, manifest and quantize.py keep their 04:21 timestamps, so later work changed only server.py. The QAT/e2e experiments were abandoned and never shipped. The trusted ternary audit's base parameter count (3,570,052,096) matches the Qwen text model.
✓text only inferenceserver.py builds the model with AutoModelForCausalLM.from_config(config.text_config), a text-only CausalLM. It loads onl
server.py builds the model with AutoModelForCausalLM.from_config(config.text_config), a text-only CausalLM. It loads only language-model tensors from weights/model.safetensors and serves text chat completions. No vision processor, visual tower or image inputs are loaded or used. quantize.py drops the visual and MTP tensors, and the trusted audit confirms model.visual.* and mtp.* are absent from storage.
✓no public benchmark eval trainingdata_manifest.json declares empty training, calibration and selection data, and quantize.py is data-free absmean PTQ tha
data_manifest.json declares empty training, calibration and selection data, and quantize.py is data-free absmean PTQ that uses no corpus. training_log.txt records only the ternary parameter count. The abandoned QAT experiments (trajectory#78, trajectory#79) used the Salesforce/wikitext wikitext-2-raw-v1 train split. That is not a public benchmark evaluation split, and those experiments are not in the shipped weights. Self-checks only send public samples to the server for gating; they are not training.
✓actual inference uses reviewed weightsserve.sh installs the bundled wheels offline and execs server.py. server.py loads weights/model.safetensors and quantiza
serve.sh installs the bundled wheels offline and execs server.py. server.py loads weights/model.safetensors and quantization_manifest.json, dequantizes each packed_2bit tensor as unpack × scale (float entries are copied directly), and assigns the results to every named parameter of the model it then serves. No other checkpoint is loaded. The trusted audit covers exactly this weights/model.safetensors file, with 0 unreferenced parameters. Per trajectory#172, the weights stayed unchanged after the copy in trajectory#121.
✓ternary executionThe trusted ternary audit passed with a ternary fraction of 0.999579: 3,569,876,992 ternary parameters against a base of
The trusted ternary audit passed with a ternary fraction of 0.999579: 3,569,876,992 ternary parameters against a base of 3,570,052,096. Only the input embedding is excluded, and it is tied to the output head. quantize.py stores codes in {-1,0,1} packed 2-bit with one bf16 scale per row. At load time server.py reconstructs every linear weight from those codes and scales, so linear-layer compute uses weights of the form ternary × scale. Norms and the embedding stay in float, as the audit allows.
✓harness credentials used only by the harnessIn the trajectory the participant used only its own allowance: the Spin compute/transfer credentials, s5cmd uploads to w
In the trajectory the participant used only its own allowance: the Spin compute/transfer credentials, s5cmd uploads to wallet storage (trajectory#82, trajectory#90) and HF_TOKEN for the pinned Qwen download (trajectory#165). No harness model API keys or base URLs were used, and no calls to claude, codex or other model APIs appear. serve.sh and server.py contain no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 5m into the runtook 16m 19s
startup 186.8 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,876,992 of 3,571,380,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999579, which must exceed 0.95
- ✓stored parameters · 3,571,380,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 186.8 s
- ✓GPQA request as graded · {"note": "grading would retry it, but the self-check had no time left; grading scores a request that still fails as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 291.4 s, median 290.3 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 2passed1h 24m into the runtook 14m 44s
startup 176.9 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,876,992 of 3,571,380,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999579, which must exceed 0.95
- ✓stored parameters · 3,571,380,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 176.9 s
- ✓GPQA request as graded · {"note": "grading would retry it, but the self-check had no time left; grading scores a request that still fails as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 300.5 s, median 299.5 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 3passed1h 47m into the runtook 15m 20s
startup 226.9 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,876,992 of 3,571,380,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999579, which must exceed 0.95
- ✓stored parameters · 3,571,380,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 226.9 s
- ✓GPQA request as graded · {"note": "grading would retry it, but the self-check had no time left; grading scores a request that still fails as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 300.3 s, median 299.2 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry || RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 4passed2h 4m into the runtook 16m 25s
startup 231.6 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 15 timed out after 300 s; 1 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry. Grading scores each such request a
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,876,992 of 3,571,380,736 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999579, which must exceed 0.95
- ✓stored parameters · 3,571,380,736 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 231.6 s
- ✓GPQA request as graded · {"note": "no reply within 300 s: grading scores a timed-out request as wrong (with no limit on how many)", "attempts": 1, "error": "RuntimeError: Submission HTTP request failed: TimeoutError: timed out"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 301.8 s, median 300.8 s; 0 stopped at max_tokens, 15 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 15 timed out after 300 s; 1 failed after up to 3 attempts: RuntimeError: Submission returned HTTP 504: b'' | no time left in the self-check for a retry. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.