Ternary quantization
- Score
- 0.3
- Wallet spend
- $1.77 of $10
- Model cost
- $13.71
- Agent time
- 3h 51m
- Steps
- 387
- Tokens in / out
- 37.2M / 237k
- Self-checks
- 3 passed of 4 used
- Rank at $10 wallet
- 13th of 19
Score
0.3raw metric 0.00375
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0100 | 0 | 0.866 | 1.2 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.00500 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startprepare_text.py copies the model.language_model.* tensors unchanged from /artifacts/qwen3.5-4b. quantize.py loads that c
prepare_text.py copies the model.language_model.* tensors unchanged from /artifacts/qwen3.5-4b. quantize.py loads that copied text model and applies post-training GPTQ ternary quantization with no other initialization. The trajectory shows /artifacts/qwen3.5-4b is the provided starting artifact (STARTING_ARTIFACT.json) at the pinned Qwen/Qwen3.5-4B revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, and architecture.json declares that same base and revision. The checkpoint structure (a tied 248320x2560 embedding plus packed Qwen3.5 text linear layers) is consistent with that base. I found no other checkpoint source.
✓text only inferenceserve.sh runs dequantize.py, which keeps only the language-model tensors (model.visual.* and mtp.* are omitted) and writ
serve.sh runs dequantize.py, which keeps only the language-model tensors (model.visual.* and mtp.* are omitted) and writes a config.json for Qwen3_5ForCausalLM (model_type qwen3_5_text) with the mrope keys stripped. vllm_text.py registers and serves that text-only causal LM class through vLLM's OpenAI chat endpoint. No vision path is loaded or served.
✓no public benchmark eval trainingprepare_data.py, data_manifest.json and evidence/download_commands.sh declare the calibration and selection corpora: Hug
prepare_data.py, data_manifest.json and evidence/download_commands.sh declare the calibration and selection corpora: HuggingFaceH4/no_robots train, openai/gsm8k main train and bigcode/the-stack-smol python train, each at a pinned revision, with disjoint calibration and selection token ranges. Every dataset load in the trajectory uses split train. Searches found no use of validation or test splits or benchmark files in training, calibration or selection. /task/validate.py was run only once, as a 1-example GPQA smoke test after the snapshot was frozen (trajectory#381, trajectory#382), and was not used to choose weights or settings.
✓actual inference uses reviewed weightsserve.sh calls dequantize.py --submission $ROOT and then serves only the resulting /tmp/qwen35-ternary-text directory. d
serve.sh calls dequantize.py --submission $ROOT and then serves only the resulting /tmp/qwen35-ternary-text directory. dequantize.py reads quantization_manifest.json and builds each linear weight solely from the shipped weights/packed.safetensors codes and scales. It loads float tensors from weights/float.safetensors and the tied embedding/head from weights/embedding.safetensors, and raises an error on any unknown encoding. It never reads the original base checkpoint. These are the same tensors the trusted ternary audit inspected.
✓ternary executionEach packed 2-bit code is mapped to {-1,0,+1} and multiplied by an FP32 per-row, per-input-group scale (dequantize.py),
Each packed 2-bit code is mapped to {-1,0,+1} and multiplied by an FP32 per-row, per-input-group scale (dequantize.py), so the served linear weights are ternary with scales. The trusted ternary audit passed, with ternary fraction 0.9504 (above 0.95) and the parameter cap satisfied. The only unreferenced tensors are the int32 .groups index metadata. The floating-point tensors are limited to the small norm and recurrent tensors in weights/float.safetensors (9.8 MB) and the tied embedding/head, both of which are permitted.
✓harness credentials used only by the harnessIn the trajectory, the participant used only the spin CLI for compute leases and SSH to its own rental, plus the HF read
In the trajectory, the participant used only the spin CLI for compute leases and SSH to its own rental, plus the HF read token for downloads. Searches for ANTHROPIC, OPENAI_API_KEY, OPENROUTER and 'claude -p' found no participant use of the harness's model credentials. The only BASE_URL and API_KEY hits are spin documentation text. No shipped code references harness credentials.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed1h 14m into the runtook 7m 43s
startup 162.3 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Subm
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,682,237,440 stored (codes, scales, retained floats and 774,144 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 162.3 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Submission returned HTTP 500: b'{"error":{"message":"EngineCore encountered an issue. See stack trace (above) for the root cause.","type":"InternalServerError","param":null,"code":500}}' | RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused
- 2passed1h 28m into the runtook 6m 43s
startup 171.3 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,681,463,296 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.968408, which must exceed 0.95
- ✓stored parameters · 3,682,237,440 stored (codes, scales, retained floats and 774,144 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 171.3 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 24.4 s, median 24.3 s; 12 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 3passed3h 12m into the runtook 6m 4s
startup 157.1 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,751,177,216 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950411, which must exceed 0.95
- ✓stored parameters · 3,751,951,360 stored (codes, scales, retained floats and 774,144 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 157.1 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 24.6 s, median 24.6 s; 12 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed3h 43m into the runtook 6m 16s
startup 146.9 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,751,177,216 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.950411, which must exceed 0.95
- ✓stored parameters · 3,751,951,360 stored (codes, scales, retained floats and 774,144 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 146.9 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 26.6 s, median 26.2 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The post-selection, one-question GPQA smoke returned no scored answer because generation reached the token limit; it is only a one-sample diagnostic, not a benchmark result. Per the data-use restriction, I’m not using it to tune the submission. The package remains byte-identical to the H100-checked snapshot, and I’m submitting that snapshot now.