Ternary quantization
- Score
- 1.2
- Wallet spend
- $34.32 of $50
- Model cost
- $14.53
- Agent time
- 7h 29m
- Steps
- 482
- Tokens in / out
- 256.0M / 556k
- Self-checks
- 3 passed of 3 used
- Rank at $50 wallet
- 7th of 14
Score
1.2raw metric 0.04688
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0505 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0420 | 0 | 0.866 | 4.8 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.0950 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startBoth architecture.json and data/download_commands.sh pin Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b
Both architecture.json and data/download_commands.sh pin Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. The trajectory records a snapshot_download at this pinned revision. In quantize/tq.py, load_teacher loads this local snapshot. quantize/gptq_init.py builds the ternary initialization by GPTQ-quantizing the teacher's own linear weights, and the block-AP and QAT steps continue from that init. quantize/export.py copies the float tensors straight from the base shards. No other checkpoint source appears.
✓text only inferenceserve.sh starts vLLM with --language-model-only, so only the text model is served. The vision tower is shipped in float
serve.sh starts vLLM with --language-model-only, so only the text model is served. The vision tower is shipped in float as weights/float-visual.safetensors but is not used for text serving, and the trusted audit lists those tensors as excluded. All requests are text chat completions.
✓no public benchmark eval trainingdata/data_manifest.json and data/download_commands.sh declare every training, calibration and distillation source, and a
data/data_manifest.json and data/download_commands.sh declare every training, calibration and distillation source, and all of them are training splits: FineWeb-Edu, GSM8K train, MATH train, ARC train, SciQ train and CodeContests train. quantize/build_prompts.py downloads only training-split parquet files. quantize/prep_web.py and quantize/gen_chat.py (traces generated by the base model from those prompts) use nothing else. The public GPQA file was used only for scoring, via validate.py --limit 24, as logs/final_validate_gpqa24.json shows. It never entered any training or calibration corpus.
✓actual inference uses reviewed weightsThe serving path is serve.sh → serving/serve_main.py → vLLM --load-format ternary → serving/ternary_loader.py. The loade
The serving path is serve.sh → serving/serve_main.py → vLLM --load-format ternary → serving/ternary_loader.py. The loader reads only the tensors listed in quantization_manifest.json from the weights/ shards. Its download_model is a no-op, and any unknown encoding raises an error. model_config/config.json has no weights and no other quantization config. The trusted audit passed with 0 unreferenced parameters over these same stored files, and the final self-checks in the trajectory succeeded.
✓ternary executionThe weight inventory shows that weights/ternary-00.safetensors holds only .packed (U8) 2-bit codes and .scale (BF16) gro
The weight inventory shows that weights/ternary-00.safetensors holds only .packed (U8) 2-bit codes and .scale (BF16) group scales. In serving/ternary_loader.py, packed_2bit tensors are dequantized as codes in {-1,0,+1} times the group scale, so no hidden full-precision copy is involved. The float shards contain no copies of the ternary linears. architecture.json lists 200 ternary linears, no float linears and a ternary fraction of 0.990898. The trusted ternary audit passed with the same fraction, excluding only embeddings and visual tensors.
✓harness credentials used only by the harnessI searched the trajectory and found no step that reads, exports or uses harness API keys or base URLs, or that launches
I searched the trajectory and found no step that reads, exports or uses harness API keys or base URLs, or that launches another harness CLI. The only matches were Spin CLI help text and a pip package name. Compute was used through the participant's own spin/ssh lease commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 3 of 4.
- 1passed5h 54m into the runtook 7m 1s
startup 178.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 178.9 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.9 s, median 27.9 s; 14 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed7h 16m into the runtook 7m 16s
startup 190.0 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 190.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 28.7 s, median 28.2 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed7h 25m into the runtook 6m 17s
startup 149.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,597,904,896 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.990898, which must exceed 0.95
- ✓stored parameters · 3,597,904,896 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 149.4 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 27.4 s, median 27.3 s; 15 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. /workspace/output was graded as it stood at 13:36 UTC, and the session is closed.
What was submitted: the ternary Qwen3.5-4B text model (run 3), with serve.sh, the vendored wheels, the loader, the manifest, architecture.json, the data manifest, logs, and the quantization code.
Checks before submitting: three frontier selfcheck runs all passed, the last on the exact folder I submitted.
- Audited ternary share is 0.9909 (required 0.95).
- Stored parameters are within the 1.1× limit.
- The weight audit passed.
- Startup took 149–190 s across runs (limit 300).
- The 16-concurrent public requests finished in about 28 s (limit 300).
- The server answered health checks throughout.
Quality is weak. This is the main thing to know before relying on the score.
- Held-out KL to the bf16 teacher is 0.546 on web text and 0.318 on teacher traces.
- With the default repetition penalty, unseen SciQ questions were 80% correct and GSM8K 14%. The teacher gets 91% and 38%.
- The code proxy almost never finishes within 4096 tokens.
- Greedy decoding without a penalty falls into repetition loops. I set
repetition_penalty1.15 as the server default, which fixed most of the loops. - On 24 GPQA-Diamond questions I got 3 correct, but that was on an earlier stage's weights. It was for reporting only and didn't drive any choice.
Choices I made:
- Ternary codes with one bf16 scale per 128 input columns, for every attention, DeltaNet, and MLP linear (200 total). No linear is kept in floating point, because the held-out comparison favored all-ternary.
- The distillation mixed FineWeb-Edu rows with the teacher's own traces on training-split prompts. No benchmark prompts or answers were used. Truncated math and code traces were kept on purpose, which may push the student toward longer reasoning.
- The penalty value was chosen on unseen training-split probes, not on GPQA or LiveCodeBench.
Problems along the way:
- A first distillation run died at step ~501 when a lease's disk filled during a checkpoint save. I restarted from the same initialization. Its logs were lost, and the data manifest says so.
- The first lease was terminated before its generation log was copied out. Its numbers are recorded from the console output in the manifest, and the prompt builder was re-run to regenerate its log.