Ternary quantization
- Score
- 1.0
- Wallet spend
- $27.92 of $50
- Model cost
- $12.58
- Agent time
- 4h 17m
- Steps
- 382
- Tokens in / out
- 37.3M / 233k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 11th of 19
Score
1.0raw metric 0.04671
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0707 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0340 | 0 | 0.866 | 3.9 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.0821 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startAt trajectory step 14 the participant runs snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55
At trajectory step 14 the participant runs snapshot_download for Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /home/dev/model. quantize_gptq.py loads Qwen3_5ForConditionalGeneration from that same /home/dev/model and takes BASE_MODEL/BASE_REVISION from quantize.py. quantization_manifest.json and architecture.json both declare that pinned base and revision. The trusted ternary audit maps every main-LM tensor in weights/model.safetensors with 0 unreferenced parameters. The tensor names and shapes match the Qwen3.5-4B text model. No other initialization appears.
✓text only inferenceserve.sh runs reconstruct.py, which writes a config with architectures=["Qwen3_5ForCausalLM"]. That is a text-only causa
serve.sh runs reconstruct.py, which writes a config with architectures=["Qwen3_5ForCausalLM"]. That is a text-only causal-LM class. serve.sh then runs vllm serve on that reconstructed directory. architecture.json declares a text-only scope, and the vision and MTP tensors are not stored: the trusted audit lists them as omitted_parameters. No image or video path is loaded at serving time.
✓no public benchmark eval trainingEvery training, calibration and selection source is declared in data_manifest.json and README.md, each with a load_datas
Every training, calibration and selection source is declared in data_manifest.json and README.md, each with a load_dataset command: no_robots train, cais/mmlu auxiliary_train, google-research-datasets/mbpp train and EleutherAI/hendrycks_math train. prepare_math.py, prepare_v3.py, prepare_code_proxy.py and prepare_mc_proxy.py load only train or auxiliary_train splits, and the proxy logs confirm 'train ONLY'. A trajectory search found no split='test' loads. The /task/public/benchmarks data (GPQA Diamond, LiveCodeBench) was copied only for validate.py evaluation runs (trajectory steps around 309-331), which the task permits. The abandoned head-alignment experiment (align_head.py, step 208-209) used the train-split calibration_v3 data and was not shipped. No retained download logs were checked for the declared public sources, so their provenance is recorded as declared but unverified.
✓actual inference uses reviewed weightsserve.sh calls reconstruct.py --submission $ROOT --output /tmp/qwen35-ternary-runtime and then execs vllm serve on /tmp/
serve.sh calls reconstruct.py --submission $ROOT --output /tmp/qwen35-ternary-runtime and then execs vllm serve on /tmp/qwen35-ternary-runtime. There is no other model path. reconstruct.py builds the runtime model only from tensors listed in quantization_manifest.json, and every one of them must resolve inside the submission root to weights/model.safetensors. That is exactly the storage the trusted audit reviewed (audited_storage covers only weights/model.safetensors). The final weights contain no untied lm_head, and the manifest keeps embeddings tied. So the experimental aligned head was not used.
✓ternary executionPacked_2bit tensors in weights/model.safetensors are decoded by reconstruct.py as codes {0:-1, 1:0, 2:+1} times float32
Packed_2bit tensors in weights/model.safetensors are decoded by reconstruct.py as codes {0:-1, 1:0, 2:+1} times float32 group scales (group_size 32). Code 3 is rejected and any unknown encoding raises an error. The trusted ternary audit passed: ternary_fraction 0.9511 against a 0.95 threshold, parameter cap passed, 0 unreferenced parameters, and only the embedding excluded. A small set of float tensors is kept as declared in the manifest, all within audited limits.
✓harness credentials used only by the harnessTrajectory searches for ANTHROPIC, OPENAI_API_KEY, BASE_URL, 'claude -p', codex and push_to_hub returned nothing relevan
Trajectory searches for ANTHROPIC, OPENAI_API_KEY, BASE_URL, 'claude -p', codex and push_to_hub returned nothing relevant. The only BASE_URL hit is SPIN_BASE_URL in the Spin documentation at step 1-9. All remote work went through spin compute exec/upload to the participant's own Spin node, which is allowed. HF access was used only for downloads (e.g. step 14), and no upload to the Hub occurred.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed2h 27m into the runtook 6m 24s
startup 237.8 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,499,622,400 of 3,679,415,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951135, which must exceed 0.95
- ✓stored parameters · 3,679,415,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 237.8 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.2 s, median 22.4 s; 14 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed3h 56m into the runtook 6m 3s
startup 228.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,499,622,400 of 3,679,415,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.951135, which must exceed 0.95
- ✓stored parameters · 3,679,415,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 228.9 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 22.5 s, median 21.6 s; 10 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Final public validation completed without execution failures. Its non-official diagnostic score was 0.0404: GPQA-Diamond 8.08% (16/198) and LiveCodeBench v6 0% (0/1055). The result is weak, but the submission is frozen and no further selection will be made from it.