Ternary quantization
- Score
- 3.5
- Wallet spend
- $13.37 of $50
- Model cost
- $1.29
- Agent time
- 6h 22m
- Steps
- 573
- Tokens in / out
- 55.9M / 461k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 7th of 19
Score
3.5raw metric 0.1039
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.207 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0460 | 0 | 0.866 | 5.3 |
| livecodebench_v6 | 0.00190 | 0 | 0.425 | 0.4 |
| mmlu_pro | 0.161 | 0.111 | 0.716 | 8.2 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0071
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned qwen starttraining/logs/download.log shows Qwen/Qwen3.5-4B downloaded at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /w
training/logs/download.log shows Qwen/Qwen3.5-4B downloaded at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /workspace/model. training/quantize.py builds the model from that directory. quantize_gptq.log reports load_state_dict missing=[] and a base perplexity of 7.64 before ternary GPTQ. QAT phase 2 resumes from the qat1 checkpoint derived from that GPTQ output (qat_finetune2.log). quantization_config.json names checkpoint qat2-900. The trajectory shows the pinned STARTING_ARTIFACT being used.
✓text only inferenceqload.py builds the model with Qwen3_5TextConfig and Qwen3_5ForCausalLM, so there is no visual tower and no MTP head. se
qload.py builds the model with Qwen3_5TextConfig and Qwen3_5ForCausalLM, so there is no visual tower and no MTP head. server.py tokenizes only the text chat messages and decodes greedily with this model. The trusted ternary audit lists no model.visual parameters in the audited storage.
✓no public benchmark eval trainingtraining/data_manifest.json and the fetch scripts declare only public train splits, each with download commands and logs
training/data_manifest.json and the fetch scripts declare only public train splits, each with download commands and logs: fineweb-edu, open-web-math, the-stack-smol, ultrachat_200k train_sft, alpaca and wikitext train. Checkpoints were selected by perplexity on held-out train-split slices. Trajectory searches found no benchmark test-split loading. The participant ran /task/validate.py on GPQA and LiveCodeBench only as plumbing and format checks, which the task instructions explicitly allow (trajectory#559). The format example in server.py was written by the participant, and no benchmark items were used for training, calibration or checkpoint selection.
✓actual inference uses reviewed weightsserve.sh runs server.py --weights $DIR with offline HF flags. server.py calls qload.load_model(args.weights), which read
serve.sh runs server.py --weights $DIR with offline HF flags. server.py calls qload.load_model(args.weights), which reads quantization_manifest.json. Every manifest entry points to weights/model-00000.safetensors, the same file the trusted audit inspected (audited_storage), with 0 unreferenced parameters. No other model or API is called during serving.
✓ternary executionThe trusted audit passed with a ternary fraction of 0.968. All linear weights are packed_2bit codes in {-1,0,+1} with fp
The trusted audit passed with a ternary fraction of 0.968. All linear weights are packed_2bit codes in {-1,0,+1} with fp16 group-32 scales. qload.py dequantizes them as W = code * scale, and quantization_config.json sets rotate to none. The only float tensor is the tied embed_tokens/lm_head, which the audit explicitly excludes.
✓harness credentials used only by the harnessTrajectory searches for API_KEY, BASE_URL, ANTHROPIC, 'claude -p' and codex found only the task and Spin documentation t
Trajectory searches for API_KEY, BASE_URL, ANTHROPIC, 'claude -p' and codex found only the task and Spin documentation text (trajectory#1). No step used or copied the harness keys or base URL, and no extra agent CLI sessions were launched. Remote GPU work used Spin over SSH (trajectory#561), which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 26m into the runtook 1m 24sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
- 2failed1h 27m into the runtook 2m 45s
startup 38.6 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "invalid literal for int() with base 10: \'input_ids\'"}}' | RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "invalid literal for int(
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 38.6 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "invalid literal for int() with base 10: \'input_ids\'"}}' | RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "invalid literal for int() with base 10: \'input_ids\'"}}' | RuntimeError: Submission returned HTTP 500: b'{"error": {"message": "invalid literal for int() with base 10: \'input_ids\'"}}'
- 3passed1h 37m into the runtook 9m 39s
startup 40.0 s of 300 s; 1/1 gates passed; WARNING concurrent requests answered: 1 well-formed replies; 15 timed out after 300 s. Grading scores each such request as wrong, so 15 of 16 here would score zero.; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones a
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 40.0 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 300.3 s, median 300.2 s; 1 stopped at max_tokens, 15 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 1 well-formed replies; 15 timed out after 300 s. Grading scores each such request as wrong, so 15 of 16 here would score zero.
- ✓healthy after requests
- 4passed5h 44m into the runtook 3m 11s
startup 35.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 35.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 41, "content_end": "The molecular symmetry group of E is C2h."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 50.5 s, median 50.5 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 5passed6h 15m into the runtook 3m 14s
startup 37.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 37.1 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 109, "content_end": "The number of stars in that region of the sky changes with distance (per unit range of distance, r) as ~ r^3."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 46.6 s, median 46.6 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 02:01:12 UTC — /workspace/output (86 files, 6.53 GB) is now being graded.
What was delivered
Weights (weights/model-00000.safetensors, 2.23 GiB): Qwen3.5-4B text tower stored as packed 2-bit ternary codes (one code per original weight) plus fp16 group-32 scales; embedding and head kept in bf16 (head tied, stored once), norms and the gated-delta-net per-channel parameters kept exact.
| audit item | value |
|---|---|
| ternary codes | 3,565,158,400 |
| fp16 group scales | 111,411,200 |
| float tensors kept exact | 4,893,696 |
| ternary share (excl. embed/head) | 0.9684 (> 0.95) |
| stored / original parameters | 1.0312 (< 1.10) |
Method: GPTQ with Hessian error compensation on a ternary grid (7.64 → 145.4 ppl), then straight-through-estimator QAT with the inference formula in the forward pass — phase 1 on raw text, phase 2 on a 31.3M-token mixture that is 58.8% chat/instruction data (ultrachat_200k train_sft, alpaca train, rendered with the base model's own template) and 41.2% fresh raw text. Exported the qat2-900 checkpoint: fineweb-edu holdout 16.73, chat holdout 5.12 (from 183.98 / 8.75 at the GPTQ start). All corpora are public training splits; selection used only held-out training-split slices.
Serving (serve.sh + server.py + qload.py): offline, vendored wheels unpacked into /tmp, 37 s to ready on the grading image, OpenAI-compatible chat endpoint with static left-padded batching (16), greedy decoding, 8192-token budget. Three decoding-side measures keep a damaged greedy model usable without touching the weights: a 4-gram repeat ban, an exact 24-token loop stop, and one generic worked example injected only when the user turn asks for an ANSWER:/\boxed{} format.
Verification: final frontier selfcheck passed (startup 37.1 s of 300 s, gate passed, all 16 concurrent requests answered well-formed); manifest re-checked independently against the safetensors headers; validate.py ran GPQA-Diamond and LiveCodeBench-v6 end to end against the served artifact.