Ternary quantization
- Score
- 0.1
- Wallet spend
- $11.51 of $50
- Model cost
- $3.40
- Agent time
- 3h 26m
- Steps
- 297
- Tokens in / out
- 37.3M / 99k
- Self-checks
- 3 passed of 4 used
- Rank at $50 wallet
- 15th of 19
Score
0.1raw metric 0.0005
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0.00200 | 0 | 0.866 | 0.2 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startThe base model Qwen/Qwen3.5-4B was downloaded at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /tmp/
The base model Qwen/Qwen3.5-4B was downloaded at the pinned revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a into /tmp/model (trajectory#34). The final checkpoint quantized_fb2 came from `layerwise4.py 50 1e-3 /tmp/model /tmp/quantized_fb2` (trajectory#257), so both the original and the work model were the pinned base. convert.py was then pointed at /tmp/quantized_fb2 to build the submission (trajectory#271). architecture.json and the manifest declare the same base and revision. The later ft.py experiments failed and were never shipped (trajectory#290-293).
✓text only inferenceserve.sh runs code/serve.py. That script loads AutoModelForCausalLM from the dequantized /tmp/model, applies the chat te
serve.sh runs code/serve.py. That script loads AutoModelForCausalLM from the dequantized /tmp/model, applies the chat template to text messages and calls model.generate. It never uses the vision tower, image processing or MTP heads. convert.py leaves the visual and mtp tensors out of the shipped weights, and the trusted ternary audit confirms they are absent.
✓no public benchmark eval trainingThe only calibration and training corpus is Salesforce/wikitext wikitext-2-raw-v1, train split. code/layerwise4.py and t
The only calibration and training corpus is Salesforce/wikitext wikitext-2-raw-v1, train split. code/layerwise4.py and training_evidence.md both state this, and the failed ft.py attempt used the same source. The trajectory shows GPQA and benchmark names only in directory listings and self-check output. No benchmark data was ever loaded for training or calibration. The public source is declared, and the download happens through load_dataset in the shipped code.
✓actual inference uses reviewed weightsserve.sh execs code/serve.py. serve.py first runs code/dequantize.py, which reads quantization_manifest.json and weights
serve.sh execs code/serve.py. serve.py first runs code/dequantize.py, which reads quantization_manifest.json and weights/model-00001.safetensors. It unpacks the 2-bit codes, multiplies by the per-row scales and writes /tmp/model/model.safetensors. serve.py then loads that same /tmp/model for generation. The trusted audit covers this same weights/model-00001.safetensors file and found 0 unreferenced tensors. In the trajectory, the converted weights were copied into /workspace/output just before submission (trajectory#272-273, #294-296).
✓ternary executionThe trusted ternary audit passed: ternary fraction 0.99941, with 3,571,184,128 stored parameters under the 3,927,057,305
The trusted ternary audit passed: ternary fraction 0.99941, with 3,571,184,128 stored parameters under the 3,927,057,305 limit. Only the input embedding (tied with the output head) is kept as float. Every targeted linear weight is stored as packed 2-bit codes in {-1,0,+1} with per-row scales (code/convert.py). The serving path rebuilds the weights only from those codes and scales (code/dequantize.py), so the model runs with ternary weights times scales.
✓harness credentials used only by the harnessHF_TOKEN was exported on the Spin GPU box only to download the model and dataset (trajectory#34, #253-259), which is all
HF_TOKEN was exported on the Spin GPU box only to download the model and dataset (trajectory#34, #253-259), which is allowed. The S3 transfers used Spin session credentials to Spin storage (trajectory#26, #272-273), not harness credentials. Searches found no model-API calls with harness keys, no copying of harness credentials and no launch of other agent CLIs. The only API_KEY and BASE_URL matches were in Spin documentation text (trajectory#13).
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed1h 35m into the runtook 11m 59s
startup 78.0 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,571,184,128 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999414, which must exceed 0.95
- ✓stored parameters · 3,571,184,128 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 78.0 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 14455, "content_end": " the the the the the the the the the the the the the the the the the the the the the the the the the the the the the the"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 291.8 s, median 291.8 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 2failed2h 54m into the runtook 6m 19s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: WARNING: Requirement '/submission/deps/*.whl' looks like a filename, but the file does not exist E
5 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,571,184,128 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999414, which must exceed 0.95
- ✓stored parameters · 3,571,184,128 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: WARNING: Requirement '/submission/deps/*.whl' looks like a filename, but the file does not exist ERROR: *.whl is not a valid wheel filename. Traceback (most recent call last): File "/submission/code/serve.py", line 1, in <module> import os, json, time, threading, queue, http.server, socketserver, torch ModuleNotFoundError: No module named 'torch'
- 3passed3h 1m into the runtook 9m 58s
startup 82.6 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,571,184,128 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999414, which must exceed 0.95
- ✓stored parameters · 3,571,184,128 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 82.6 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 12250, "content_end": "the and the and the and the and the and the and the and the and the and the and the and the and the and the"}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 274.2 s, median 274.2 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
- 4passed3h 15m into the runtook 11m 18s
startup 91.4 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,571,184,128 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.999414, which must exceed 0.95
- ✓stored parameters · 3,571,184,128 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 91.4 s
- ✓GPQA request as graded · {"finish_reason": "stop", "answer": null, "note": "no answer letter found (ANSWER: X on its own line): grading scores it wrong", "content_characters": 4126, "content_end": " "}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 264.9 s, median 264.9 s; 0 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submission complete.