Ternary quantization
- Score
- 0.0
- Wallet spend
- $0.00 of $10
- Model cost
- $15.06
- Agent time
- 7h 5m
- Steps
- 325
- Tokens in / out
- 76.9M / 312k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 11th of 14
Score
0.0raw metric 0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0 | 0 | 0.866 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe trajectory shows STARTING_ARTIFACT is Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. gptq_ter
The trajectory shows STARTING_ARTIFACT is Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. gptq_ternary.py loads that local base checkpoint and asserts that only lm_head may be missing. build_submission.py quantizes the original tensors and copies the non-target language tensors unchanged into float.safetensors. It writes the base and revision into quantization_manifest.json. The trajectory also checks the manifest against the original tensor headers. I found no different initialization.
✓text only inferenceserver.py builds Qwen3_5TextConfig from config_text.json (qwen3_5_text, 32 layers, tied embeddings) and instantiates Qwe
server.py builds Qwen3_5TextConfig from config_text.json (qwen3_5_text, 32 layers, tied embeddings) and instantiates Qwen3_5ForCausalLM. It serves through a text chat endpoint. The vision and MTP tensors are left out of the shipped weights, and the trusted audit confirms the visual and MTP towers are omitted.
✓no public benchmark eval trainingCalibration used 16×1024 tokens from the train splits of fineweb-edu, bigcode/starcoderdata and HuggingFaceTB/smoltalk.
Calibration used 16×1024 tokens from the train splits of fineweb-edu, bigcode/starcoderdata and HuggingFaceTB/smoltalk. These sources are declared in make_calib.py, calib_manifest.json and DATA_SOURCES.md, and the run is visible at trajectory#150. Benchmark files (gpqa_diamond.csv, livecodebench test.jsonl) were only opened to check their prompt format (trajectory#203). They were not used for calibration, training or selection.
✓actual inference uses reviewed weightsserve.sh runs server.py offline. server.py rebuilds every manifest tensor: weights/ternary.safetensors codes times weigh
serve.sh runs server.py offline. server.py rebuilds every manifest tensor: weights/ternary.safetensors codes times weights/scales.safetensors group scales, plus the float tensors from weights/float.safetensors. It then loads them into the model and ties the embeddings. The trusted audit references these same shipped files and reports 0 unreferenced parameters. The final listing (trajectory#323) shows these files and nothing else.
✓ternary executionThe trusted ternary audit passed, with ternary_fraction 0.9593 and the storage cap satisfied. All 200 text linear layers
The trusted ternary audit passed, with ternary_fraction 0.9593 and the storage cap satisfied. All 200 text linear layers are stored as packed 2-bit codes in {-1,0,+1} with per-group bf16 scales (group size 8/16/64 by layer). server.py reconstructs them as codes × scale. server.py has an optional salient float-column override, but weights/salient.safetensors is not shipped (weight inventory, trajectory#323, METHOD.md), so that path is dormant. Only embed_tokens (tied to lm_head) and the norm/small tensors stay in float.
✓harness credentials used only by the harnessSearches of the trajectory for API keys, base URLs, ANTHROPIC, OPENROUTER, printenv and curl found no use of harness cre
Searches of the trajectory for API keys, base URLs, ANTHROPIC, OPENROUTER, printenv and curl found no use of harness credentials. The only external storage action was a Spin s5cmd upload to the participant's own allowed storage (trajectory#317), followed by submission (trajectory#324).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 18m into the runtook 10m 55s
startup 129.2 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 129.2 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 301.2 s, median 300.2 s; 0 stopped at max_tokens, 16 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests
- 2passed6h 2m into the runtook 12m 48s
startup 245.6 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,716,279,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.959335, which must exceed 0.95
- ✓stored parameters · 3,716,279,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 245.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 18084, "content_end": " of a\nLet m know that the question is a question about the product of a\nLet m know that the question is a question about"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 301.1 s, median 300.1 s; 0 stopped at max_tokens, 16 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 0 well-formed replies; 16 timed out after 300 s. Grading scores each such request as wrong, so 16 of 16 here would score zero.
- ✓healthy after requests