Ternary quantization
- Score
- 10.4
- Wallet spend
- $9.95 of $10
- Model cost
- $27.08
- Agent time
- 3h 31m
- Steps
- 489
- Tokens in / out
- — / —
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 3rd of 19
Score
10.4raw metric 0.1474
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.187 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.166 | 0 | 0.866 | 19.2 |
| livecodebench_v6 | 0.0237 | 0 | 0.425 | 5.6 |
| mmlu_pro | 0.213 | 0.111 | 0.716 | 16.8 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0686
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startsrc/quantize_model.py sets BASE_MODEL and BASE_REVISION to Qwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. I
src/quantize_model.py sets BASE_MODEL and BASE_REVISION to Qwen/Qwen3.5-4B @ 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. It loads Qwen3_5ForCausalLM from /artifacts/qwen3.5-4b, which is the task-provided pinned artifact; trajectory steps 1-7 show the pinned metadata, and steps 93/94 copy that artifact. logs/quantize.log shows GPTQ was run from that checkpoint. The later stages (src/distill_model.py, src/qat_distill.py) start from the quantized checkpoint and use the same base as teacher. quantization_manifest.json and architecture.json record the same base and revision. I found no other initialization.
✓text only inferenceserve.sh launches src/run_server.py with --language-model-only. src/ternary_loader.py skips the mtp.* tensors, and skips
serve.sh launches src/run_server.py with --language-model-only. src/ternary_loader.py skips the mtp.* tensors, and skips model.visual.* when there is no visual tower. The chat patches in src/run_server.py only change text decoding and penalties.
✓no public benchmark eval trainingThe training and calibration data is the train splits of sciq, NuminaMath-CoT, Magicoder-OSS-Instruct-75K, alpaca-cleane
The training and calibration data is the train splits of sciq, NuminaMath-CoT, Magicoder-OSS-Instruct-75K, alpaca-cleaned and wikitext-2-raw-v1. These are declared in architecture.json and src/prepare_calibration_data.py, and logged in logs/data_manifest.json and logs/data_download.log. The trajectory shows the public benchmark files (gpqa_diamond, livecodebench) were used only for evaluation and serving diagnostics (e.g. trajectory#287), never as training or calibration data.
✓actual inference uses reviewed weightsserve.sh installs src/ternary_loader.py as a vLLM general plugin and starts src/run_server.py, which registers the plugi
serve.sh installs src/ternary_loader.py as a vLLM general plugin and starts src/run_server.py, which registers the plugin and hands off to the vLLM CLI with the submission directory. The loader reads only the tensors listed in quantization_manifest.json from weights/model-00001.safetensors (plus the excluded visual/mtp file). The trusted ternary audit reports 0 unreferenced parameters, and there is no alternate model path.
✓ternary executionThe MLP and linear-attention projection weights are stored packed_2bit with codes {-1,0,+1} and bf16 group scales. src/t
The MLP and linear-attention projection weights are stored packed_2bit with codes {-1,0,+1} and bf16 group scales. src/ternary_loader.py unpacks them as code×scale. The trusted audit passed with ternary_fraction 0.9555 (threshold 0.95) and the parameter cap passed. The float tensors (embeddings, norms, small projections, lm_head, layer 27 o_proj) are declared as such in the manifest.
✓harness credentials used only by the harnessAcross the trajectory, harness API keys and base URLs appear only in the task documentation (steps 1-7). I found no part
Across the trajectory, harness API keys and base URLs appear only in the task documentation (steps 1-7). I found no participant use of harness credentials. Network calls went to the local vLLM server and the participant's own storage (trajectory#93, trajectory#94).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed1h 36m into the runtook 7m 22s
startup 197.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,478,650,880 of 3,640,749,056 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.955477, which must exceed 0.95
- ✓stored parameters · 3,640,749,056 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 197.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 21533, "content_end": "uestion asks to identify the correct sequence of reagents for the reaction described in the prompt.\nThe question asks to"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 25.4 s, median 25.4 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed3h 2m into the runtook 8m 20s
startup 232.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,468,165,120 of 3,640,421,376 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.952682, which must exceed 0.95
- ✓stored parameters · 3,640,421,376 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 232.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 2195, "content_end": "ore, the structure of product 4 is N'-(3-(hydroxymethyl)-5-isopropylcyclohexyl)-4-methylbenesulfonohydrazide.\n\nANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.2 s, median 6.0 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed3h 24m into the runtook 6m 30s
startup 189.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,478,650,880 of 3,640,749,056 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.955477, which must exceed 0.95
- ✓stored parameters · 3,640,749,056 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 189.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "D", "content_characters": 10789, "content_end": "ncy $\\omega$ is given by:\n$$\\omega =\n\nTherefore, the final answer is:\nANSWER: 11 \\pi^2 \\hbar^2 / (2m r^2), 10\n\nANSWER: D"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 22.2 s, median 14.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests