Trial · ternary-ptq-50s3ay

Ternary quantization

Qwen3.8 MaxxhighHarness Claude Code 2.1.291

Wallet

Score: 2.5
Wallet spend: $8.91 of $10
Model cost: $36.84
Agent time: 6h
Steps: 375
Tokens in / out: 37.4M / 332k
Self-checks: 2 passed of 2 used
Rank at $10 wallet: 6th of 14

Loading trajectory…

Score

2.5raw metric 0.03034

The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.

Benchmarks

Benchmark	Raw	Floor	Ceiling	Normalized
gpqa_diamond	0.00505	0.250	0.571	0.0
math_500	0.0780	0	0.866	9.0
livecodebench_v6	0.00474	0	0.425	1.1
mmlu_pro	0.0336	0.111	0.716	0.0

Verifier details

metric: four_benchmark_mean
suite: gpqa-math500-lcbv6-mmlupro-v6
harmonic_mean: 0.0089

pass6 of 6 checks passed · reviewed by claude-opus-5-5 · read 375 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓pinned qwen startThe trajectory shows the participant downloading the base on the Spin GPU with `hf download Qwen/Qwen3.5-4B --revision 8

The trajectory shows the participant downloading the base on the Spin GPU with `hf download Qwen/Qwen3.5-4B --revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a`, which is the pinned revision. In src/model_util.py, load_text_model loads model.language_model.* from that artifact's safetensors index. The student starts as TernarizedLinear, which takes its latents and least-squares scales from those original weights. src/kd.py and the recon step then refine this student. finalize.log reports a rel-L2 of about 0.44–0.47 against the original weights, which fits a model derived from that base. architecture.json declares the same base and revision.

architecture.jsonsrc/model_util.pysrc/kd.pyevidence/logs/finalize.logtrajectory#130

✓text only inferenceserve.sh runs `vllm serve /tmp/serve-model ... --language-model-only`. serving/build_serving_dir.py builds a text-only c

serve.sh runs `vllm serve /tmp/serve-model ... --language-model-only`. serving/build_serving_dir.py builds a text-only checkpoint: the vision tower and MTP are not stored, and the trusted audit confirms both are omitted. No image or multimodal path is served.

serve.shserving/build_serving_dir.py

✓no public benchmark eval trainingevidence/data_manifest.json and src/make_calib.py use only train splits: HuggingFaceFW/fineweb-edu sample-10BT, codeparr

evidence/data_manifest.json and src/make_calib.py use only train splits: HuggingFaceFW/fineweb-edu sample-10BT, codeparrot/github-code-clean, open-web-math and wikimedia/wikipedia 20231101.en. The load commands are recorded and the calib log matches the trajectory output. The held-out eval slice is cut from the same train data. The participant opened /task/public/benchmarks only to look at it, and no benchmark data appears in the calibration, recon or KD code (src/kd.py trains on the calib token memmap).

evidence/data_manifest.jsonsrc/make_calib.pyevidence/logs/calib.logsrc/kd.pytrajectory#301trajectory#303

✓actual inference uses reviewed weightsserve.sh runs serving/build_serving_dir.py. That script reads quantization_manifest.json and weights/{codes,scales,liter

serve.sh runs serving/build_serving_dir.py. That script reads quantization_manifest.json and weights/{codes,scales,literals,embed,head}.safetensors, which are the files the trusted audit covered. It writes /tmp/serve-model/model.safetensors, and vllm then serves that directory. serve.sh sets HF_HUB_OFFLINE=1 and uses no other source of weights.

serve.shserving/build_serving_dir.pyquantization_manifest.jsonweights/codes.safetensorsweights/scales.safetensorsweights/literals.safetensorsweights/embed.safetensorsweights/head.safetensors

✓ternary executionIn quantization_manifest.json, every large linear (MLP, attention q/k/v/o, and linear-attention in_proj_qkv/in_proj_z/ou

In quantization_manifest.json, every large linear (MLP, attention q/k/v/o, and linear-attention in_proj_qkv/in_proj_z/out_proj) is stored as packed 2-bit ternary codes with group-32 scales. build_serving_dir.py dequantizes each one as scale*(code-1). The trusted audit passed with a ternary fraction of 0.9684, the parameter cap met and no unreferenced parameters. Only the embedding and the untied lm_head are excluded as float.

serving/build_serving_dir.pyquantization_manifest.jsonweights/codes.safetensorsweights/scales.safetensors

✓harness credentials used only by the harnessThe only credential the participant used was HF_TOKEN, and only for model and dataset downloads on Spin, which is allowe

The only credential the participant used was HF_TOKEN, and only for model and dataset downloads on Spin, which is allowed. Searches of the trajectory found no use of ANTHROPIC or OPENAI keys, no harness base URL, no openrouter calls by the participant, no harness CLI launches and no uploads. serve.sh uses no credentials.

trajectory#301serve.sh

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.

1
failed1h 39m into the runtook 3m 11sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2
passed4h 10m into the runtook 10m 1s
startup 157.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 157.9 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 157.1 s, median 157.1 s; 14 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
3
passed5h 54m into the runtook 6m 18s
startup 115.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,681,463,296 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 115.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 1594, "content_end": "metry is a fundamental symmetry that must be respected by all operators.\n\nTherefore, the correct answer is **A. 1,2,4**."}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 103.2 s, median 103.2 s; 13 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests