Ternary quantization
- Score
- 41.2
- Wallet spend
- $192.65 of $200
- Model cost
- $42.30
- Agent time
- 7h 46m
- Steps
- 790
- Tokens in / out
- 78.3M / 802k
- Self-checks
- 2 passed of 2 used
- Rank at $200 wallet
- 2nd of 14
Score
41.2raw metric 0.3379
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.126 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.658 | 0 | 0.866 | 76.0 |
| livecodebench_v6 | 0.194 | 0 | 0.425 | 45.8 |
| mmlu_pro | 0.373 | 0.111 | 0.716 | 43.2 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.2316
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json names Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In the trajecto
architecture.json names Qwen/Qwen3.5-4B at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In the trajectory, snapshot_download fetches that model at that revision. code/model.py loads the text config and weights from the downloaded directory. Per README.md and code/train.py, the ternary weights are initialised by Lloyd alternation from the original weights and distilled from the frozen original model. The trusted ternary audit lists every model.language_model.* tensor and omits only the visual and MTP tensors, which fits a Qwen3.5-4B-derived text model. No other initialisation was found.
✓text only inferencecode/load_ternary.py skips every tensor whose name starts with model.visual. or mtp. serve.sh launches the vLLM OpenAI s
code/load_ternary.py skips every tensor whose name starts with model.visual. or mtp. serve.sh launches the vLLM OpenAI server with --language-model-only, so serving is text only.
✓no public benchmark eval trainingdata_manifest.json and evidence/data/provenance_parts declare only train splits, each with a pinned source (e.g. gsm8k,
data_manifest.json and evidence/data/provenance_parts declare only train splits, each with a pinned source (e.g. gsm8k, hendrycks_math, apps, code_contests, arc, mmlu auxiliary_train, medqa/medmcqa train, sciq, openbookqa, openr1). The build scripts open train files only: build_mcq.py, build_code.py ('train split only (valid/test never opened)') and build_math.py ('TRAIN splits only'). GPQA and LCB were used only to decontaminate the data and as dev measurements. Checkpoint selection used training-split proxies. Downloads for some declared public sources are unverified, but every source is declared.
✓actual inference uses reviewed weightsserve.sh runs code/load_ternary.py on the submission. The loader reads quantization_manifest.json and weights/model.safe
serve.sh runs code/load_ternary.py on the submission. The loader reads quantization_manifest.json and weights/model.safetensors, dequantises the weights into /tmp/submission_model, and vLLM is then started on exactly that directory. The trusted audit's audited_storage covers the weights/model.safetensors language-model tensors with 0 unreferenced tensors. Only the embedding is excluded, which the policy permits.
✓ternary executionPacked 2-bit codes (-1/0/+1) with bf16 group scales (group size 32) are served as code × scale. The trusted ternary audi
Packed 2-bit codes (-1/0/+1) with bf16 group scales (group size 32) are served as code × scale. The trusted ternary audit passed: ternary fraction 0.9684 and stored size 3,681,463,296, under the 3,927,057,305 limit. The high-precision tensors left are the excluded embedding and the norm/small tensors the audit allows.
✓harness credentials used only by the harnessNo SDK, CLI or BASE_URL use of the harness credentials appears in the shipped code or the trajectory. trajectory#766 and
No SDK, CLI or BASE_URL use of the harness credentials appears in the shipped code or the trajectory. trajectory#766 and trajectory#767 are only leak scans: the values are redacted and the grep found no matches. The credentials were not used for model calls or copied into the submission.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 56m into the runtook 4m 33s
startup 80.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,625,757,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.983286, which must exceed 0.95
- ✓stored parameters · 3,625,757,696 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 80.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 25.0 s, median 24.5 s; 14 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed7h 42m into the runtook 5m 7s
startup 74.7 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 74.7 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 23.9 s, median 23.9 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in and the session is over. The final self-check passed (startup 74.7 s of 300 s, ternary fraction 0.9684, stored parameters 1.0312× the original non-embedding count, weight audit passed).
The submitted weights are the step-2499 export of the ternary Qwen3.5-4B student, trained by distillation from the original model.
Checkpoint choice used only a held-out proxy built from training-split data (STEM multiple-choice, MATH train, Codeforces train), with the rule fixed before the last candidate was run. The step-2499 export scored 0.467 mean proxy accuracy against 0.442 for step 2000. It truncates slightly more often than step 2000 on all three kinds of problem.
I also ran the GPQA-Diamond and LiveCodeBench dev files through validate.py, after the choice was made and as measurement only:
| GPQA-Diamond dev | LiveCodeBench dev | |
|---|---|---|
| step 2000 | 0.157 | 0.162 |
| submitted (step 2499) | 0.086 | 0.194 |
The submitted export is worse on GPQA, and I kept it anyway, because the rules forbid selecting on the public validation files. The main weakness is answers cut off at the 4096-token limit: 142 of 198 GPQA dev answers came back empty.
Distilling the teacher onto the student's own greedy generations made truncation worse in code (69% at step 2493). The final 99 steps without that data recovered it. No ablation was run to separate the effect of that data from the extra annealed steps.
The README in /workspace/output/ reports all of this, including the negative results.