Ternary quantization
- Score
- 0.1
- Wallet spend
- $8.63 of $200
- Model cost
- $0.93
- Agent time
- 2h 6m
- Steps
- 515
- Tokens in / out
- 66.1M / 365k
- Self-checks
- 0 passed of 4 used
- Rank at $200 wallet
- 12th of 14
Score
0.1raw metric 0.004454
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0.0101 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.00200 | 0 | 0.866 | 0.2 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0.00571 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. quantize.py reads model
architecture.json declares Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. quantize.py reads model.language_model.* tensors from the downloaded base checkpoint and writes the shipped literal, ternary and scales safetensors. train_ternary.py then fine-tunes starting from that quantized checkpoint, which it loads through model_loader. The conversion and training logs match this lineage, and no other initialization appears anywhere.
✓text only inferenceserve.sh installs the local wheels and runs server.py. server.py tokenizes text-only chat prompts and calls model.genera
serve.sh installs the local wheels and runs server.py. server.py tokenizes text-only chat prompts and calls model.generate on input_ids alone. model_loader.py builds AutoModelForCausalLM from config.text_config. There is no vision tower or MTP path, and the manifest holds language-model tensors only.
✓no public benchmark eval trainingThe only training and calibration corpus is Salesforce/wikitext wikitext-2-raw-v1 train. It is declared in logs/data_sou
The only training and calibration corpus is Salesforce/wikitext wikitext-2-raw-v1 train. It is declared in logs/data_sources.json, with its download command and sha256 retained, and the trajectory shows the WikiText download and its use in calibration and training. GPQA Diamond was used only in the post-freeze limit-8 validation run, as recorded in logs/final_validation_summary.json and training_evidence.json. No source code reads benchmark data for training, calibration or selection.
✓actual inference uses reviewed weightsmodel_loader.py opens only the shipped weights/literal.safetensors, weights/ternary.safetensors and weights/scales.safet
model_loader.py opens only the shipped weights/literal.safetensors, weights/ternary.safetensors and weights/scales.safetensors through quantization_manifest.json. It loads with strict key checks: any missing key other than the tied lm_head, or any unexpected key, raises an error. The server uses this loader, and the trusted ternary audit covers these same weight files.
✓ternary executionThe large linear matrices are stored as packed 2-bit codes (−1, 0, +1) with one scale per group of 32. The loader recons
The large linear matrices are stored as packed 2-bit codes (−1, 0, +1) with one scale per group of 32. The loader reconstructs each weight as code × scale. Only declared float exceptions, embeddings, norms and small tensors stay in float. The trusted ternary audit passed with a ternary fraction of 0.9539, and stored parameters are within the limit.
✓harness credentials used only by the harnessThe trajectory's only credential exposure is a redacted env listing (trajectory#6), which is allowed. Searches found no
The trajectory's only credential exposure is a redacted env listing (trajectory#6), which is allowed. Searches found no use or copying of OPENAI_API_KEY or BASE_URL values, no direct OpenAI API calls and no model sub-sessions. Remote work went through the permitted Spin CLI.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed15m 46s into the runtook 36sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
- 2failed16m 59s into the runtook 6m 18s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: r /submission/requirements.txt (line 37)) Processing ./wheels/requests-2.34.2-py3-none-any.whl (fr
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: r /submission/requirements.txt (line 37)) Processing ./wheels/requests-2.34.2-py3-none-any.whl (from -r /submission/requirements.txt (line 38)) Processing ./wheels/urllib3-2.8.0-py3-none-any.whl (from -r /submission/requirements.txt (line 39)) Processing ./wheels/certifi-2026.7.22-py3-none-any.whl (from -r /submission/requirements.txt (line 40)) Processing ./wheels/charset_normalizer-3.5.2-cp312-cp312-manylinux2014_x86_64.manylinux_2_17_x86_64.manylinux_2_28_x86_64.whl (from -r /submission/requirements.txt (line 41)) Installing collected packages: mpmath, urllib3, typing-extensions, typer, truststore, triton, transformers, tqdm, torch, tokenizers, sympy, shellingham, setuptools, safetensors, rich, requests, regex, PyYAML, Pygments, packaging, numpy, networkx, mdurl, MarkupSafe, markdown-it-py, Jinja2, idna, huggingface-hub, httpx2, httpcore2, hf-xet, fsspec, flash-linear-attention, fla-core, filelock, einops, click, charset-normalizer, certifi, anyio, annotated-doc Successfully installed Jinja2-3.1.6 MarkupSafe-3.0.4 PyYAML-6.0.3 Pygments-2.21.0 annotated-doc-0.0.5 anyio-4.15.1 certifi-2026.7.22 charset-normalizer-3.5.2 click-8.5.0 einops-0.8.2 filelock-4.0.12 fla-core-0.5.2 flash-linear-attention-0.5.2 fsspec-2026.9.0 hf-xet-1.7.0 httpcore2-2.13.1 httpx2-2.13.1 huggingface-hub-2.2.0 idna-3.20 markdown-it-py-4.2.0 mdurl-0.1.2 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 packaging-26.3 regex-2026.9.29 requests-2.34.2 rich-15.0.0 safetensors-0.8.0 setuptools-84.0.0 shellingham-1.5.4 sympy-1.14.0 tokenizers-0.23.2 torch-2.8.0+cu128 tqdm-4.70.1 transformers-5.19.0 triton-3.4.0 truststore-0.10.4 typer-0.27.3 typing-extensions-4.16.0 urllib3-2.8.0 Traceback (most recent call last): File "/submission/server.py", line 15, in <module> import torch File "/tmp/qwen35-ternary-runtime/tor ...[clipped]
- 3failed24m 33s into the runtook 6m 35s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: , filelock, einops, click, charset-normalizer, certifi, anyio, annotated-doc Successfully installe
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: , filelock, einops, click, charset-normalizer, certifi, anyio, annotated-doc Successfully installed Jinja2-3.1.6 MarkupSafe-3.0.4 PyYAML-6.0.3 Pygments-2.21.0 annotated-doc-0.0.5 anyio-4.15.1 certifi-2026.7.22 charset-normalizer-3.5.2 click-8.5.0 einops-0.8.2 filelock-4.0.12 fla-core-0.5.2 flash-linear-attention-0.5.2 fsspec-2026.9.0 hf-xet-1.7.0 httpcore2-2.13.1 httpx2-2.13.1 huggingface-hub-2.2.0 idna-3.20 markdown-it-py-4.2.0 mdurl-0.1.2 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 regex-2026.9.29 requests-2.34.2 rich-15.0.0 safetensors-0.8.0 setuptools-84.0.0 shellingham-1.5.4 sympy-1.14.0 tokenizers-0.23.2 torch-2.8.0+cu128 tqdm-4.70.1 transformers-5.19.0 triton-3.4.0 truststore-0.10.4 typer-0.27.3 typing-extensions-4.16.0 urllib3-2.8.0 /tmp/qwen35-ternary-runtime/torch/cuda/__init__.py:182: UserWarning: CUDA initialization: Unexpected error from cudaGetDeviceCount(). Did you run some cuda functions before calling NumCudaDevices() that might have already set an error? Error 803: system has unsupported display driver / cuda driver combination (Triggered internally at /pytorch/c10/cuda/CUDAFunctions.cpp:109.) return torch._C._cuda_getDeviceCount() > 0 Traceback (most recent call last): File "/submission/server.py", line 236, in <module> main() File "/submission/server.py", line 225, in main model, tokenizer = load_model_and_tokenizer() ...[clipped]
- 4failed31m 40s into the runtook 6m 51s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: usolver-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, nvidia
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: usolver-cu12, nvidia-curand-cu12, nvidia-cufile-cu12, nvidia-cufft-cu12, nvidia-cudnn-cu12, nvidia-cuda-runtime-cu12, nvidia-cuda-nvrtc-cu12, nvidia-cuda-cupti-cu12, nvidia-cublas-cu12, numpy, networkx, mdurl, MarkupSafe, markdown-it-py, Jinja2, idna, huggingface-hub, httpx2, httpcore2, hf-xet, fsspec, flash-linear-attention, fla-core, filelock, einops, click, charset-normalizer, certifi, anyio, annotated-doc Successfully installed Jinja2-3.1.6 MarkupSafe-3.0.4 PyYAML-6.0.3 Pygments-2.21.0 annotated-doc-0.0.5 anyio-4.15.1 certifi-2026.7.22 charset-normalizer-3.5.2 click-8.5.0 einops-0.8.2 filelock-4.0.12 fla-core-0.5.2 flash-linear-attention-0.5.2 fsspec-2026.9.0 hf-xet-1.7.0 httpcore2-2.13.1 httpx2-2.13.1 huggingface-hub-2.2.0 idna-3.20 markdown-it-py-4.2.0 mdurl-0.1.2 mpmath-1.3.0 networkx-3.7 numpy-2.5.3 nvidia-cublas-cu12-12.8.4.1 nvidia-cuda-cupti-cu12-12.8.90 nvidia-cuda-nvrtc-cu12-12.8.93 nvidia-cuda-runtime-cu12-12.8.90 nvidia-cudnn-cu12-9.10.2.21 nvidia-cufft-cu12-11.3.3.83 nvidia-cufile-cu12-1.13.1.3 nvidia-curand-cu12-10.3.9.90 nvidia-cusolver-cu12-11.7.3.90 nvidia-cusparse-cu12-12.5.8.93 nvidia-cusparselt-cu12-0.7.1 nvidia-nccl-cu12-2.27.3 nvidia-nvjitlink-cu12-12.8.93 nvidia-nvtx-cu12-12.8.90 packaging-26.3 regex-2026.9.29 requests-2.34.2 rich-15.0.0 safetensors-0.8.0 setuptools-84.0.0 shellingham-1.5.4 sympy-1.14.0 tokenizers-0.23.2 torch-2.8.0+cu128 tqdm-4.70.1 transformers-5.19.0 triton-3.4.0 truststore-0.10.4 typer-0.27.3 typing-extensions-4.16.0 urllib3-2.8.0 Traceback (most recent call last): File "/submission/server.py", line 236, in <module> main() File "/submission/server.py", line 225, in main model, tokenizer = load_model_and_tokenizer() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/submission/model_loader.py", line 109, in load ...[clipped]
- 5failed39m 3s into the runtook 3m 27s
startup 76.9 s of 300 s; failed GPQA request as graded: RuntimeError: RuntimeError: Submission returned HTTP 400: b'{"error":{"message":"\'tokenizers.Encoding\' object is not subscriptable","type":"invalid_request_error","param":null,"code":null}}' | RuntimeError: Submission returned HTTP 40
6 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,681,586,176 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.969444, which must exceed 0.95
- ✓stored parameters · 3,681,586,176 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 76.9 s
- ✗GPQA request as graded · RuntimeError: RuntimeError: Submission returned HTTP 400: b'{"error":{"message":"\'tokenizers.Encoding\' object is not subscriptable","type":"invalid_request_error","param":null,"code":null}}' | RuntimeError: Submission returned HTTP 400: b'{"error":{"message":"\'tokenizers.Encoding\' object is not subscriptable","type":"invalid_request_error","param":null,"code":null}}' | RuntimeError: Submission returned HTTP 400: b'{"error":{"message":"\'tokenizers.Encoding\' object is not subscriptable","type":"invalid_request_error","param":null,"code":null}}'
The agent's last message before its session ended.
The final weights pass a local manifest audit: all 426 language tensors are represented, ternary storage is 95.39% when scales are counted as non-ternary, and stored parameter-equivalent size is 3.07% above the original. The exact final package also reached /health on the H100. I’m doing the final file and evidence checks before submitting.