Ternary quantization
- Score
- 0.3
- Wallet spend
- $8.91 of $10
- Model cost
- $0.86
- Agent time
- 4h 23m
- Steps
- 334
- Tokens in / out
- 31.7M / 353k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 14th of 19
Score
0.3raw metric 0.03827
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0556 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.00800 | 0 | 0.866 | 0.9 |
| livecodebench_v6 | 0.000948 | 0 | 0.425 | 0.2 |
| mmlu_pro | 0.0886 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0.0033
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
The score comes from a regrade that re-scored this same submission without reviewing it again; this is the review from its original grading (2026-10-21).
✓pinned qwen startarchitecture.json and quantization_manifest.json name Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd
architecture.json and quantization_manifest.json name Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. In trajectory#3, /task/public/STARTING_ARTIFACT.json gives that same repo and revision. Every GPTQ and QAT run (trajectory steps 81-98, 99-114, 154-179 and 196-218) passes --model-dir .../models--Qwen--Qwen3.5-4B/snapshots/851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. evidence/qat.py and evidence/gptq_cal.py read the base model.language_model.* shards from that directory and initialize the ternary weights from them using sequential GPTQ, then QAT. The shipped manifest's 426 language tensors match that structure. No other initialization was found.
✓text only inferenceserver.py builds a Qwen3_5TextConfig and a Qwen3_5ForCausalLM (text only) on the meta device. It loads only manifest ent
server.py builds a Qwen3_5TextConfig and a Qwen3_5ForCausalLM (text only) on the meta device. It loads only manifest entries named model.language_model.* and ties lm_head to embed_tokens. No vision or MTP modules are built and no image inputs are handled. serve.sh just runs server.py offline.
✓no public benchmark eval trainingevidence/data_manifest.json declares every corpus along with its download command. The QAT and GPTQ calibration data is
evidence/data_manifest.json declares every corpus along with its download command. The QAT and GPTQ calibration data is the train split of wikitext-103-raw-v1, codeparrot-clean, open-web-math and fineweb-edu sample-10BT. Chat repair uses the ultrachat_200k train_sft split. Checkpoint selection uses fineweb-edu train rows from 40000 on. evidence/prep_train.py, prep_chat.py and mkeval.py load only those train splits. In the trajectory, /task/public/benchmarks was only listed. The 'Generating test/validation split' lines are automatic load_dataset output, and the code selects split='train'. The prompt-format probes (trajectory step 195) used hand-written questions, and I found no benchmark prompts or test/validation data used for training, calibration or selection. Raw download logs are kept only as the prep logs, so the provenance of the declared sources is not fully verified. That alone does not violate the check.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs server.py. load_model r
serve.sh installs the vendored wheels offline, sets HF_HUB_OFFLINE/TRANSFORMERS_OFFLINE and runs server.py. load_model reads quantization_manifest.json and fills every parameter from weights/quant-0000[0-3].safetensors (packed codes plus scales) and weights/float.safetensors. Any missing or mismatched shape raises an error. These are the same files listed in the trusted ternary audit's audited_storage, which reported 0 unreferenced parameters. The serving path loads no alternate or original checkpoint.
✓ternary executionPacked_2bit tensors are unpacked into codes {-1,0,+1} and multiplied by an fp16 scale per group of 32 (dequant_weight in
Packed_2bit tensors are unpacked into codes {-1,0,+1} and multiplied by an fp16 scale per group of 32 (dequant_weight in server.py). Literal float copies of quantized tensors are rejected. The trusted audit passed: ternary_fraction 0.968 and the parameter cap met, with only the tied embedding excluded as float, which the task allows.
✓harness credentials used only by the harnessThe trajectory (steps 1-334) shows the participant using only participant-facing tools. The spin CLI was used for comput
The trajectory (steps 1-334) shows the participant using only participant-facing tools. The spin CLI was used for compute, including `spin transfer credentials` (trajectory#124), which spin documents as 'export temporary S3 credentials for a VM or another client', to move its own artifacts through wallet-scoped R2 storage. HF_TOKEN was used only to download models and datasets, as the task allows. Reading the validator's judge code (trajectory steps 11-25) was inspection only: I found no use of a judge key, the harness's model API key or any other harness credential. serve.sh contains no credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed2h 56m into the runtook 8m 14s
startup 58.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 58.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 34.9 s, median 17.7 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2passed3h 30m into the runtook 8m 15s
startup 50.8 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 50.8 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 488, "content_end": "thyl 3-(2-((2,4-dimethylphenyl)sulfinyl)ethyl)-2-oxocycolhexane-1-caraxylate (or D)\n\nThe final answer is A or B, C or D."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 280.1 s, median 280.1 s; 8 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed3h 52m into the runtook 9m 39s
startup 64.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 64.9 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 669, "content_end": "ylpent=4>-enoic acids\n\nReason through the question, then put your final answer on its own line as ANWER: A (or B, C, D)."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 281.2 s, median 280.2 s; 8 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed4h 12m into the runtook 7m 45s
startup 50.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 50.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 193, "content_end": " luminosity of Star_1 compared to Star_2, and the options provided are A, B, C, D. The correct answer is A (or B, C, D)."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 270.1 s, median 254.8 s; 7 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted.
Final artifact (/workspace/output, 87 files, 6.41 GB) — all four self-checks passed:
| Gate | Result |
|---|---|
| Ternary fraction | 96.84% (required >95%) |
| Stored parameters | 3.681e9 vs 3.927e9 limit |
| Weight audit | inspect_ternary passed |
| Startup | 50.4 s (required 300) |
| 16 concurrent requests | slowest 270.1 s, 0 timed out, 16 well-formed |
What shipped
- 200 quantized linears as LSB-first
packed_2bitcodes with per-group-32 fp16 scales; 226 remaining tensors (norms,conv1d,in_proj_a/b,A_log,dt_bias, tied embedding/head) as float. 426 manifest entries covering every original language tensor. - Pipeline: sequential GPTQ init → QAT stage 1 (scales + float params, frozen codes; ppl 273 → 25.4) → QAT stage 2 with 50% ultrachat train_sft data for chat-format repair. All corpora are train splits; provenance and download commands in
evidence/data_manifest.json. - Serving: vendored cp312 wheels installed offline into
/tmp/qwen_env, static batching up to 16, chat-template prompting with an instruction-style fallback when the model would answer with EOS, repetition/no-repeat-ngram penalties, 2048-token output ceiling and a 270 s answer guard.
Last changes this session (all validated by the final self-check): replaced 5 ms polling with per-request Event waits, batched the sampling syncs (2 per step instead of 2×B), raised the listen backlog to 256 so 16 simultaneous connections are never refused, and capped generation at 2048 tokens. I also tested flash-linear-attention — logits matched exactly but it gave no measurable speedup, since decode time is dispatch-bound, so it was not shipped.