Ternary quantization
- Score
- 24.7
- Wallet spend
- $9.03 of $10
- Model cost
- $41.57
- Agent time
- 5h 43m
- Steps
- 807
- Tokens in / out
- 82.4M / 737k
- Self-checks
- 3 passed of 3 used
- Rank at $10 wallet
- 1st of 14
Score
24.7raw metric 0.2412
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.167 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.420 | 0 | 0.866 | 48.5 |
| livecodebench_v6 | 0.0853 | 0 | 0.425 | 20.1 |
| mmlu_pro | 0.293 | 0.111 | 0.716 | 30.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.1701
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares the base as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trainin
architecture.json declares the base as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The training pipeline (tq.py, blockap.py, e2e.py, export_ckpt.py) starts from the provided base artifact at /artifacts/qwen3.5-4b and quantizes it. The trajectory shows the task-provided starting artifact (STARTING_ARTIFACT.json) being read and used. The checkpoint structure matches Qwen3.5-4B: language linears are ternary, embeddings are tied, and the visual and mtp tensors are retained. No other initialization was found.
✓text only inferenceserve.sh launches vLLM with --language-model-only. convert_weights.py skips the model.visual. and mtp. prefixes when it
serve.sh launches vLLM with --language-model-only. convert_weights.py skips the model.visual. and mtp. prefixes when it builds the serving directory. policy_proxy.py forwards only text chat requests to the local vLLM.
✓no public benchmark eval trainingdata_manifest.json and logs/download_commands.sh declare only train splits: OpenThoughts3, OpenScienceReasoning-2, ultra
data_manifest.json and logs/download_commands.sh declare only train splits: OpenThoughts3, OpenScienceReasoning-2, ultrachat_200k train_sft, tulu-3-sft-mixture and FineWeb-edu. The proxy dev sets come from train splits (train/proxy/README.md). prep_prompts.py reads only the train parquets and applies a 12-gram decontamination filter. build_decontam_ref.py reads the public GPQA/LCB examples only to build that filter's reference, and logs/prep_prompts.log shows contaminated prompts being dropped. In the trajectory, the public benchmarks were used only for small validate.py smoke tests (8 GPQA items), not for training, calibration or selection. Some download logs are retained but not independently verified.
✓actual inference uses reviewed weightsserve.sh runs convert_weights.py on the shipped weights directory and then serves only the resulting bf16 directory with
serve.sh runs convert_weights.py on the shipped weights directory and then serves only the resulting bf16 directory with vLLM. No other model path or download exists in the serving path. The packed_2bit tensors and scales in weights/model-00001.safetensors, together with the embeddings in weights/model-00002.safetensors, are the ones listed in quantization_manifest.json and audited by the trusted ternary audit (0 unreferenced, 0 omitted).
✓ternary executionAll language linear weights are stored as packed_2bit ternary codes (-1/0/+1) with per-group fp16 scales. convert_weight
All language linear weights are stored as packed_2bit ternary codes (-1/0/+1) with per-group fp16 scales. convert_weights.py dequantizes them exactly as code times scale; that is a lossless representation of the ternary weights, with no extra float residuals. The trusted ternary audit passed, with ternary_fraction 0.9572 and the parameter cap passed.
✓harness credentials used only by the harnessSearches of the trajectory for API keys, Anthropic/OpenAI endpoints, claude -p and printenv found no use of harness cred
Searches of the trajectory for API keys, Anthropic/OpenAI endpoints, claude -p and printenv found no use of harness credentials by participant code. API_KEY matches are only Spin documentation text, validator CLI help (an unused judge key option) and a self-audit grep. The trajectory shows only the harness's own Agent subagent calls and Spin CLI compute usage, both of which are allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1failed1h 37m into the runtook 1m 21sfree (stopped before any GPU time)
failed review prerequisites: The review needs architecture.json and serve.sh at the top of the submission
1 gates
- ✗review prerequisites · The review needs architecture.json and serve.sh at the top of the submission
- 2passed1h 39m into the runtook 4m 45s
startup 126.3 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 126.3 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 25.6 s, median 25.6 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed4h 16m into the runtook 5m 29s
startup 124.8 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,724,717,056 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.957162, which must exceed 0.95
- ✓stored parameters · 3,724,717,056 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 124.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 11120, "content_end": "**?\n\n(I am almost out of space, so I stop here and state my final answer now, in the format requested above.)\n\nANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.3 s, median 26.1 s; 8 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed5h 39m into the runtook 4m 46s
startup 103.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,724,717,056 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.957162, which must exceed 0.95
- ✓stored parameters · 3,724,717,056 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 103.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "C", "content_characters": 10862, "content_end": " ~4\n\n(I am almost out of space, so I stop here and state my final answer now, in the format requested above.)\n\nANSWER: C"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.2 s, median 25.6 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The ternary Qwen3.5-4B is submitted: frontier selfcheck passed on the final files, then frontier submit --confirm went through at 05:32 UTC.
I could not run my own accuracy check on the final weights. The 4090 box's GPU failed and the A100 was down. The checks I do have: the export audit and the selfcheck gates. The held-out KL against the original model also fell from 0.461 after Block-AP to 0.289, which is how I chose this checkpoint.
Selfcheck results:
- Ternary fraction: 0.9572, against the required 0.95.
- Stored parameters: 3,724,717,056, under the 1.1× limit of about 3.93 billion.
- Startup:
/healthanswered 200 after 103.7 s (limit 300 s). - Concurrent requests: with 16 at once, the slowest took 26.2 s and none timed out.
- Not covered: selfcheck reports no score, and its note says the architecture review did not run and can still score a submission zero.
Serving runs vLLM on the dequantized weights, with a small proxy in front that turns thinking off. It reserves the last 384 tokens (640 for programming prompts) to force a final answer. Earlier tests of the original model scored 0.675 this way versus 0.196 with native thinking, which truncates most answers at 4096 tokens.