Ternary quantization
- Score
- 0.0
- Wallet spend
- $5.76 of $10
- Model cost
- $0.54
- Agent time
- 58m
- Steps
- 296
- Tokens in / out
- 37.6M / 240k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 11th of 14
Score
0.0raw metric 0
The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.
Benchmarks
| Benchmark | Raw | Floor | Ceiling | Normalized |
|---|---|---|---|---|
| gpqa_diamond | 0 | 0.250 | 0.571 | 0.0 |
| math_500 | 0 | 0 | 0.866 | 0.0 |
| livecodebench_v6 | 0 | 0 | 0.425 | 0.0 |
| mmlu_pro | 0 | 0.111 | 0.716 | 0.0 |
Verifier details
- metric
- four_benchmark_mean
- suite
- gpqa-math500-lcbv6-mmlupro-v6
- harmonic_mean
- 0
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned qwen startarchitecture.json declares the base as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. code/quanti
architecture.json declares the base as Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a. code/quantize.py loads Qwen3_5ForConditionalGeneration from the local copy at /root/qwen with that revision pinned, and evidence/download_base_model.sh and evidence/data_provenance.json show the download at that revision. Quantization does no gradient updates; it only applies GPTQ-style ternary rounding plus float exceptions taken from the original weights. code/train_head.py uses the same pinned BF16 model as the teacher. The only trained part is an untied lm_head, which starts from embed_tokens, and the policy allows that. The trajectory shows the final quantization rerun from /root/qwen on the Spin VM (trajectory#157, trajectory#174).
✓text only inferenceserve.sh runs code/serve.py, which loads only Qwen3_5ForCausalLM through code/model_loader.py, keeps only the text parts
serve.sh runs code/serve.py, which loads only Qwen3_5ForCausalLM through code/model_loader.py, keeps only the text parts of each message, renders them with the chat template and calls model.generate. The vision tower is omitted; the trusted audit lists model.visual.* as omitted. No image, audio or video path exists.
✓no public benchmark eval trainingQuantization calibration (code/quantize.py) and head distillation (code/train_head.py) both load Salesforce/wikitext wik
Quantization calibration (code/quantize.py) and head distillation (code/train_head.py) both load Salesforce/wikitext wikitext-103-v1 with split=train only. This is declared in quantization_manifest.json, evidence/data_provenance.json, evidence/head_calibration.json and evidence/head_calibration.log (split=train, 65,536 tokens). The trajectory shows GPQA and LiveCodeBench rows from /task/public/benchmarks copied into a five-row validation sample, but they were used only to score the server through validate.py, never for training, calibration or selection (trajectory#173, trajectory#176). The note in data_provenance.json says the HF builder cached all splits but only train is read; the code confirms this.
✓actual inference uses reviewed weightsserve.sh sets MODEL_DIR to the submission root and execs code/serve.py. That calls load_model, which reads every tensor
serve.sh sets MODEL_DIR to the submission root and execs code/serve.py. That calls load_model, which reads every tensor referenced by quantization_manifest.json from weights/ternary.safetensors, weights/scales.safetensors and weights/floating.safetensors. load_state_dict is strict about missing and unexpected keys, so nothing is silently left out. The trusted audit reports zero unreferenced parameters and no uninspected files. The trajectory shows these exact weight files moved from the VM into the output directory (trajectory#226, trajectory#227).
✓ternary executionLinear and conv weights are stored as 2-bit packed codes in {-1,0,+1} with FP16 group scales. code/model_loader.py dequa
Linear and conv weights are stored as 2-bit packed codes in {-1,0,+1} with FP16 group scales. code/model_loader.py dequantizes them as (code-1)*scale, with no other added terms. The float tensors (embed_tokens, lm_head, norms, small SSM parameters and a budgeted set of about 120M exception parameters) are declared float in quantization_manifest.json. The trusted ternary audit passed with ternary_fraction 0.952 and parameter_cap_passed=true.
✓harness credentials used only by the harnessThe participant environment held only HF_TOKEN and the Spin broker variables (trajectory#7), and both uses are allowed.
The participant environment held only HF_TOKEN and the Spin broker variables (trajectory#7), and both uses are allowed. The "codex exec" process seen in trajectory#55 is ps output showing the harness's own process, not a call made by the participant. I found no participant use of OpenAI or other harness API keys or endpoints, and code/serve.py makes no external API calls.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed47m 40s into the runtook 12m 22s
startup 99.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,450,601,472 of 3,623,967,744 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.952161, which must exceed 0.95
- ✓stored parameters · 3,623,967,744 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 99.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 296.3 s, median 295.2 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 06:32 UTC.
The audit found 95.216% ternary parameters outside the embedding and output head, with stored body parameters at 101.51% of the original count. frontier selfcheck passed: startup was 99.2 seconds, and the slowest of 16 concurrent requests took 296.3 seconds against the 300-second limit.
The limited validator sample scored 0/5 on both GPQA and LiveCodeBench; completions were cut off at the token limit. I did not use those samples to tune the model.
Key files: serve.sh, quantization_manifest.json, data_provenance.json, and quantize.py.