Ternary quantization
- Score
- 1.4
- Wallet spend
- $60.41 of $200
- Model cost
- $29.46
- Agent time
- 7h 5m
- Steps
- 356
- Tokens in / out
- 74.2M / 203k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 10th of 14
Score
1.4raw metric 0.04289
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0253 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0380 | 0 | 0.866 | 4.4 |
| livecodebench_v6 | 0.00474 | 0 | 0.425 | 1.1 |
| mmlu_pro | 0.104 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0140
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In trajectory step 152
architecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In trajectory step 152 the participant downloaded the model with `hf download Qwen/Qwen3.5-4B --revision 851bf6e8...`. code/run_gptq.py loads that local copy (/home/Ubuntu/work/qwen35-4b) and quantizes from it with GPTQ. code/pack_output.py builds the shipped weights from the GPTQ/BRECQ/E2E output plus the same original model directory. No other starting checkpoint appears anywhere.
✓text only inferenceserve.sh runs build_served_model.py and then the plain vLLM OpenAI API server with --enforce-eager. There is no speculat
serve.sh runs build_served_model.py and then the plain vLLM OpenAI API server with --enforce-eager. There is no speculative or MTP flag and no multimodal input setup. The visual and MTP tensors are kept as unchanged float tensors but nothing serving uses them. The selection evals in the trajectory sent only text chat messages.
✓no public benchmark eval trainingevidence/DATA.md declares that every training, calibration and selection corpus comes from a public train split: smollm-
evidence/DATA.md declares that every training, calibration and selection corpus comes from a public train split: smollm-corpus fineweb-edu-dedup, smol-smoltalk, OpenR1-Math-220k, sciq, ai2_arc, openbookqa, Magicoder-Evol-Instruct and gsm8k. The data scripts all call load_dataset with split="train". Teacher responses came from the original model answering those train prompts. The selection eval (prep_eval_prompts2.py, trajectory step 299) also used only train splits. Public GPQA items were used only for an 8-item serving smoke test (trajectory steps 258-261), not for training or selection. Download logs were not retained, so the declared provenance is recorded as unverified.
✓actual inference uses reviewed weightsserve.sh calls code/build_served_model.py. That script reads quantization_manifest.json and reconstructs the served mode
serve.sh calls code/build_served_model.py. That script reads quantization_manifest.json and reconstructs the served model only from weights/lang.safetensors, embed.safetensors, visual.safetensors and mtp.safetensors. vLLM then serves that reconstructed directory. These are the same files listed in the trusted audit's audited_storage and in the weight inventory. No other model weights are shipped or downloaded at serve time. The only uninspected file is a Triton kernel cache .so.
✓ternary executionEach packed_2bit tensor is unpacked to codes in {-1,0,+1} and multiplied by its group-of-64 fp32 scale. Inference theref
Each packed_2bit tensor is unpacked to codes in {-1,0,+1} and multiplied by its group-of-64 fp32 scale. Inference therefore uses exactly scale*code. The only float tensors are the ones the manifest marks as float: embedding, norms, conv1d, A_log/dt_bias, in_proj_a/b, and the unused visual and MTP tensors. The trusted ternary audit passed with a ternary fraction of 0.9833 and no unreferenced tensors. The stored-parameter check in trajectory step 317 reported 101.56% of the original size.
✓harness credentials used only by the harnessThe trajectory shows no use of harness or OpenRouter API keys. HF_TOKEN was used only to download the model and datasets
The trajectory shows no use of harness or OpenRouter API keys. HF_TOKEN was used only to download the model and datasets (trajectory steps 152 and 301). The Spin S3 session credentials (trajectory step 249) are the participant's own transfer storage, not harness credentials. The frontier selfcheck and submit commands are the intended harness interfaces.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 51m into the runtook 12m 6s
startup 216.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 216.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 32, "content_end": "<think>\n<think>\n<think>\n<think>\n"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 146.2 s, median 1.8 s; 1 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed6h 20m into the runtook 10m 3s
startup 210.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 210.6 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 137.3 s, median 137.3 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
repetition_penalty doesn't fix the conceptual loops and worsens correct answers — excluding it. v2 is final (validated by selfcheck #2 with this exact tree). Submitting.