Ternary quantization
- Score
- 0.5
- Wallet spend
- $8.09 of $10
- Model cost
- $37.24
- Agent time
- 2h 41m
- Steps
- 102
- Tokens in / out
- 55.5M / 177k
- Self-checks
- 1 passed of 2 used
- Rank at $10 wallet
- 9th of 14
Score
0.5raw metric 0.004
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.0160 | 0 | 0.866 | 1.8 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe shipped weights come from post-training quantization of the provided Qwen3.5-4B checkpoint. No training was done. qu
The shipped weights come from post-training quantization of the provided Qwen3.5-4B checkpoint. No training was done. quantize.py and the packer that was actually run (pack_legal_gpu.py) read the original shards through model.safetensors.index.json. They write base_model Qwen/Qwen3.5-4B with base_revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a, which matches architecture.json. The trajectory shows the packer running on the H100 against /workspace/model (step 58). The packed files went into the output folder through Spin S3 (steps 61-71). Step 74 checked that the manifest covers all 426 language tensors of the /artifacts/qwen3.5-4b index. The trusted audit finds a consistent tensor structure with the model's real shapes.
✓text only inferenceserver.py loads only Qwen3_5ForCausalLM through load_model.py, using the text config (model_type qwen3_5_text) and the t
server.py loads only Qwen3_5ForCausalLM through load_model.py, using the text config (model_type qwen3_5_text) and the tokenizer. It serves text chat completions only. The visual.* and MTP tensors are left out of the manifest and are never loaded, and the trusted audit lists them as omitted.
✓no public benchmark eval trainingNo benchmark data was used for calibration or selection. The quantizer picks its scales from the weights alone, scoring
No benchmark data was used for calibration or selection. The quantizer picks its scales from the weights alone, scoring several support quantiles, and reads no corpus (quantize.py, pack_legal_gpu.py in trajectory step 47). Layouts were compared using a few FineWeb sample-10BT train-split sequences, which is an allowed training split declared in data_manifest.json and download_log.txt. The download command is recorded but there is no retained output log, so that source is unverified. Layouts were also compared on generic prompts such as 'capital of France' and '17*23'. /task/public/benchmarks was only listed (step 8). The only public benchmark inputs sent were those of the harness self-check, which is evaluation, and the final layout was chosen for audit legality, not benchmark results (step 98).
✓actual inference uses reviewed weightsserve.sh runs server.py, which calls load_ternary_text_model(QUANT_DIR) with QUANT_DIR defaulting to the submission dire
serve.sh runs server.py, which calls load_ternary_text_model(QUANT_DIR) with QUANT_DIR defaulting to the submission directory. load_model.py iterates quantization_manifest.json and reads weights/packed.safetensors, weights/scales.safetensors and weights/floats.safetensors. It raises an error on missing tensors or an unknown encoding. These are exactly the files the trusted audit reviewed: fraction 0.9983, 0 unreferenced tensors, no uninspected files. No other checkpoint is loaded.
✓ternary executionEvery quantized nn.Linear is replaced by TernaryLinear. Its weight is rebuilt from the packed 2-bit codes as scale*(code
Every quantized nn.Linear is replaced by TernaryLinear. Its weight is rebuilt from the packed 2-bit codes as scale*(code-1), giving values of -1, 0 or +1 times one per-row fp16 scale (ternary_linear.py, load_model.py). The trusted audit passed with 3,565,158,400 ternary parameters out of the audited total (fraction 0.998313) and within the parameter cap. Only the small tensors and the embedding (tied to the head) stay in float, as allowed.
✓harness credentials used only by the harnessThe trajectory shows only allowed credential use: Spin compute, Spin s5cmd/R2 transfers within the wallet prefix, and a
The trajectory shows only allowed credential use: Spin compute, Spin s5cmd/R2 transfers within the wallet prefix, and a read-only HF token on the leased machine. Searches found no direct use of API keys or base URLs, no Anthropic or OpenAI calls, and no Hugging Face uploads. The 'upload' calls were spin compute upload and s5cmd copies to the wallet's own storage (steps 7, 47, 61-62, 70).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1failed1h 57m into the runtook 1m 35sfree (stopped before any GPU time)
failed ternary fraction: 3,565,158,400 of 5,018,397,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.710418, which must exceed 0.95
2 gates
- ✓review prerequisites
- ✗ternary fraction · 3,565,158,400 of 5,018,397,696 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.710418, which must exceed 0.95
- 2failed2h 27m into the runtook 6m 45s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 19, in <module> from f
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,571,182,592 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.998313, which must exceed 0.95
- ✓stored parameters · 3,571,182,592 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: Traceback (most recent call last): File "/submission/server.py", line 19, in <module> from fastapi import FastAPI, Request File "/opt/venv/lib/python3.12/site-packages/fastapi/__init__.py", line 7, in <module> from .applications import FastAPI as FastAPI File "/opt/venv/lib/python3.12/site-packages/fastapi/applications.py", line 7, in <module> from fastapi import routing File "/opt/venv/lib/python3.12/site-packages/fastapi/routing.py", line 58, in <module> from fastapi.dependencies.utils import ( File "/opt/venv/lib/python3.12/site-packages/fastapi/dependencies/utils.py", line 52, in <module> from fastapi.background import BackgroundTasks File "/opt/venv/lib/python3.12/site-packages/fastapi/background.py", line 5, in <module> from fastapi.telemetry._api import _operation File "/opt/venv/lib/python3.12/site-packages/fastapi/telemetry/__init__.py", line 1, in <module> from ._api import TelemetryConfig as TelemetryConfig File "/opt/venv/lib/python3.12/site-packages/fastapi/telemetry/_api.py", line 9, in <module> from opentelemetry import context as otel_context ModuleNotFoundError: No module named 'opentelemetry'
- 3passed2h 35m into the runtook 7m 35s
startup 132.0 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,565,158,400 of 3,571,182,592 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.998313, which must exceed 0.95
- ✓stored parameters · 3,571,182,592 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 132.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 35, "content_end": "0.\n\n0 20\n\n2 \n\n\n 21,\n 1/b\n\n1/\n ** \n"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 207.9 s, median 9.5 s; 1 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in. Qwen3.5-4B is packed as one ternary code per weight, with one scale per output row, and the graded self-check passed.
The weight audit counts every scale as a non-ternary parameter. Group scales, which are what actually keep this model coherent, drop the ternary fraction to 0.71. One scale per row passes at 0.9983 and stays inside the storage cap. On the grading image the server became healthy in 132 seconds, and 16 concurrent requests finished in 208 seconds.
That layout does not preserve accuracy. The weights are nearly Gaussian, so a single scale per row has about 20% relative error, and that error compounds across 32 layers. Public GPQA and LiveCodeBench replies in the self-check were not answers. Layouts that do answer need either group scales or more than one exact layer, and both fail the 95% rule.