Ternary quantization
- Score
- 6.4
- Wallet spend
- $49.72 of $200
- Model cost
- $1.52
- Agent time
- 9h 19m
- Steps
- 653
- Tokens in / out
- 61.3M / 514k
- Self-checks
- 3 passed of 3 used
- Rank at $200 wallet
- 9th of 19
Score
6.4raw metric 0.1285
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.207 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.140 | 0 | 0.866 | 16.2 |
| livecodebench_v6 | 0.00474 | 0 | 0.425 | 1.1 |
| mmlu_pro | 0.162 | 0.111 | 0.716 | 8.4 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0175
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned qwen startThe starting artifact is Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (trajectory#3). train_qat.
The starting artifact is Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a (trajectory#3). train_qat.py fills the fp32 latent student through load_checkpoint_weights from /workspace/model, using the model.language_model.* keys of the original HF shards. The teacher is the same frozen checkpoint. export_ckpt.py exports every model.language_model.* tensor under its original name and raises KeyError if one is missing. quantization_manifest.json records base_model Qwen/Qwen3.5-4B and the pinned base_revision. The trusted ternary audit matches the pinned inventory: 4,659,865,088 inventory parameters, no unreferenced tensors, and only the vision tower and MTP head omitted.
✓text only inferenceqwen35.py implements only the text decoder (GatedDeltaNet and gated attention layers, with a tied head). It has no visio
qwen35.py implements only the text decoder (GatedDeltaNet and gated attention layers, with a tied head). It has no vision or MTP modules. server.py and tokenizer_util.py accept only string message content on the OpenAI chat endpoint. The vision tower and MTP head are not shipped (see the trusted audit's omitted_parameters).
✓no public benchmark eval trainingdata_manifest.json and prep_data.py declare only public training corpora: ultrachat_200k train_sft, NuminaMath-CoT, WebI
data_manifest.json and prep_data.py declare only public training corpora: ultrachat_200k train_sft, NuminaMath-CoT, WebInstructSub, OpenCodeReasoning split_0, fineweb-edu sample-10BT and wikitext-103 train. prep_mc.py and logs/prep_mc.log use only train splits: ARC train, OpenBookQA train, MMLU auxiliary_train, HellaSwag train, CommonsenseQA train and SciQ train. pick_variant.py chooses the variant by NLL on a held-out tail of those training corpora. The trajectory shows validate.py runs on /task/public/benchmarks only as development diagnostics, as the task allows (for example trajectory#547). No benchmark prompts or answers were added to any training or calibration data, and no validation or test split was loaded.
✓actual inference uses reviewed weightsserve.sh runs server.py --output-dir with the submission directory. server.py calls load_packed_model from loader.py, wh
serve.sh runs server.py --output-dir with the submission directory. server.py calls load_packed_model from loader.py, which goes through manifest['tensors'] in quantization_manifest.json. It reads each listed tensor from weights/model-00000.safetensors or weights/model-00001.safetensors, unpacks packed_2bit entries, and copies them into the model with shape checks. GraphDecoder captures decode on that same model object, and no other weight source is opened at serve time. The trusted audit reports zero unreferenced parameters.
✓ternary executionexport_ckpt.py and weights_io.py store 200 linear tensors as 2-bit codes (-1/0/+1) with group-128 absmean scales. loader
export_ckpt.py and weights_io.py store 200 linear tensors as 2-bit codes (-1/0/+1) with group-128 absmean scales. loader.py's unpack_ternary rebuilds each weight as code × scale, so the served linear layers compute exactly with ternary-valued weights times per-group scales. The trusted audit passed with ternary_fraction 0.9909 (threshold 0.95) and stored parameters within the 1.1× cap. The only full-precision tensors are the tied embedding/head, norms, conv1d and the small in_proj_a/b gates.
✓harness credentials used only by the harnessIn the trajectory, the participant only read help text for the spin and openrouter CLIs and grepped the validator's --ju
In the trajectory, the participant only read help text for the spin and openrouter CLIs and grepped the validator's --judge-key-env option (trajectory#3, trajectory#396). No OpenRouter key was created and no judge key or harness credential was used anywhere. HF_TOKEN was used only for downloading models and datasets, as permitted. No shipped file references harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed59m 39s into the runtook 5m 47s
startup 57.5 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 57.5 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 78.4 s, median 78.4 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed9h 4m into the runtook 3m 48s
startup 54.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 54.7 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 9, "content_end": "ANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 63.0 s, median 62.9 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed9h 13m into the runtook 3m 24s
startup 46.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 46.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": "A", "content_characters": 9, "content_end": "ANSWER: A"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 61.0 s, median 61.0 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted. Final state of /workspace/output:
Weights (ternary-QAT, packed 2-bit)
- 426 language tensors; 3,565,158,400 ternary codes with absmean group-128 scales (27.85M scale values, counted as non-ternary).
- Ternary fraction of audited parameters excluding the embedding/head: 0.99090 (packed codes counted four per byte; 0 bad codes in the audit).
- Stored non-embedding parameters 3,597,904,896 vs. the 1.1× limit of 3,927,057,305 — 91.6% of the allowance, exactly one code per original weight, no stacked stages.
- 2.29 GB in two shards;
quantization_manifest.jsoncarriesQwen/Qwen3.5-4B@851bf6e8…, and every original LM tensor name.
Recovery
- Straight-through QAT of fp32 latents with KL distillation from the frozen bf16 checkpoint, then a short answer-format stage: 17.0M tokens of public multiple-choice training splits (ARC/OpenBookQA/MMLU-aux/HELLASWAG/CommonsenseQA), shuffled lettered options,
ANSWER: <letter>targets, mixed 50/50 with a 17.0M-token sample of the main corpus. - Held-out multiple-choice parse rate went 0/24 → 24/24 (the pre-stage export replied
The correct answer is $\boxed{\text{C}}$, which the harness does not extract); held-out MC NLL 2.43 → 1.44. - General held-out NLL 1.80 → 1.84 (float model 1.59), so the format gain cost ~0.04 nat; own-prompt probes still solve algebra, probability, and emit fenced Python.
Serving
serve.shinstalls the vendored unmodified PyPI/PyTorch wheels into/tmp, then runs a batched engine with CUDA-graph decode, 8192-token context, and a length guard that closes long replies withfinish_reason: "stop".- Live pod run: ready in 5.4 s, health 200, 16/16 concurrent requests HTTP 200 at 572 tok/s warm (192 tok/s cold, median 10.2 s), 4,439-token prompt answered in 0.4 s. Logs are in
logs/serve.logandlogs/bench.log. - Three grading selfchecks passed, the last on the exact submitted folder: startup 46.1 s of 300 s, 1/1 gates passed.
Provenance, download commands, training logs (including the abandoned phase-3 restart), probe samples, and all evidence files referenced by architecture.json are present and resolve.