Trial · ternary-ptq-2qopo5

Ternary quantization

Muse Spark 1.3maxHarness Muse Code 1.4.0-R4161.1

Wallet

Score: 9.3
Wallet spend: $70.84 of $200
Model cost: $20.63
Agent time: 11h 50m
Steps: 359
Tokens in / out: 83.7M / 316k
Self-checks: 2 passed of 2 used
Rank at $200 wallet: 5th of 14

Loading trajectory…

Score

9.3raw metric 0.09431

The raw metric (accuracy on GPQA Diamond, MATH-500, LiveCodeBench v6 and MMLU-Pro) is rescaled so that chance on each benchmark scores 0 and the original bf16 Qwen3.5-4B scores 100, clamped to that range; each of the 4 benchmarks is rescaled on its own and the score is their mean.

Benchmarks

Benchmark	Raw	Floor	Ceiling	Normalized
gpqa_diamond	0	0.250	0.571	0.0
math_500	0.148	0	0.866	17.1
livecodebench_v6	0.00853	0	0.425	2.0
mmlu_pro	0.221	0.111	0.716	18.1

Verifier details

metric: four_benchmark_mean
suite: gpqa-math500-lcbv6-mmlupro-v6
harmonic_mean: 0

pass6 of 6 checks passed · reviewed by claude-opus-5-5 · read 359 trajectory steps

After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.

✓pinned qwen startPass. The trajectory shows the participant downloading Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8c

Pass. The trajectory shows the participant downloading Qwen/Qwen3.5-4B at revision 851bf6e806efd8d0a36b00ddf55e13ccb7b8cd0a with `hf download`. evidence/code/train.py loads both the student and the frozen teacher from that HF snapshot path, and evidence/train_run.json records the same path. Training ran as QAT from that snapshot. evidence/code/export.py rebuilds the base model, converts it to ternary, loads qat1/ckpt_002699.pt with strict=True, and writes the packed codes, scales and float tensors. The manifest's ternary ratio (98.435%) matches the trusted audit. quantization_manifest.json records the same base model and revision. No other checkpoint or initialization appears anywhere.

evidence/code/train.pyevidence/code/export.pyevidence/train_run.jsonquantization_manifest.jsontrajectory#34trajectory#275

✓text only inferencePass. server.py serves an OpenAI-compatible chat endpoint. It renders messages with tok.apply_chat_template and calls mo

Pass. server.py serves an OpenAI-compatible chat endpoint. It renders messages with tok.apply_chat_template and calls model.generate on input_ids only; pixel values or images are never passed. The vision-tower weights are loaded as unused float parameters, and the MTP head is omitted, per the audit.

server.pyserve.sh

✓no public benchmark eval trainingPass. evidence/code/data.py and evidence/data_manifest.json declare the training data: train splits of public pretrainin

Pass. evidence/code/data.py and evidence/data_manifest.json declare the training data: train splits of public pretraining and instruction corpora (fineweb-edu sample/100BT, smollm-corpus cosmopedia-v2, finemath-4plus, smoltalk all, starcoderdata python), each at a pinned revision. prep_data.py and prep_code.py retain the download commands. The trajectory shows the task's public benchmark files were only listed and viewed. The diagnostic and probe prompts were the participant's own, and the checkpoint was chosen by the final step and training loss, not by benchmark score.

evidence/data_manifest.jsonevidence/code/data.pyevidence/code/prep_data.pyevidence/code/prep_code.pyevidence/README.mdtrajectory#3

✓actual inference uses reviewed weightsPass. serve.sh execs server.py, which calls loader.load_state_dict(ROOT). That reads quantization_manifest.json and buil

Pass. serve.sh execs server.py, which calls loader.load_state_dict(ROOT). That reads quantization_manifest.json and builds tensors only from weights/ternary-00001.safetensors, weights/float-00001.safetensors and weights/float-00002.safetensors. The model is loaded with strict=False, but the server raises if anything other than the tied lm_head.weight is missing. The trusted ternary audit passed with 0 unreferenced tensors. The trajectory confirms the final export was copied to output, and that later edits changed only server.py (startup speed and a logits processor), never the weights.

serve.shserver.pyloader.pyquantization_manifest.jsonweights/float-00001.safetensorsweights/float-00002.safetensorsweights/ternary-00001.safetensorstrajectory#356

✓ternary executionPass. For every packed_2bit tensor, loader.py unpacks 2-bit codes to {-1,0,1} and multiplies by per-group (size 64) bf16

Pass. For every packed_2bit tensor, loader.py unpacks 2-bit codes to {-1,0,1} and multiplies by per-group (size 64) bf16 scales. Only tensors the manifest declares as float (embeddings, norms, linear_attn A_log/conv1d/dt_bias, visual) stay float. The trusted audit passed with a ternary fraction of 0.98435, and stored size is within the 1.1x limit. Dequantising once at startup to bf16 still executes the ternary values times their scales.

loader.pyquantization_manifest.jsonweights/ternary-00001.safetensorsweights/float-00001.safetensors

✓harness credentials used only by the harnessPass. Searching the trajectory for harness credential variables and endpoints (ANTHROPIC, OPENROUTER, BASE_URL, API_KEY)

Pass. Searching the trajectory for harness credential variables and endpoints (ANTHROPIC, OPENROUTER, BASE_URL, API_KEY) found no use. The participant used only spin compute and frontier selfcheck/submit tooling, and made no LLM API calls with harness credentials.

trajectory#1trajectory#357

LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.

frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.

1
failed10h 14m into the runtook 2m 10sfree (stopped before any GPU time)
failed weight audit: The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2 gates
- ✓review prerequisites
- ✗weight audit · The weight audit rejected quantization_manifest.json or its tensors: ValueError: Nonternary literal code in model.language_model.embed_tokens.weight
2
passed10h 17m into the runtook 16m 43s
startup 291.8 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,625,819,136 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.984354, which must exceed 0.95
- ✓stored parameters · 3,625,819,136 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 291.8 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 291.4 s, median 290.3 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests
3
passed11h 33m into the runtook 14m 35s
startup 152.2 s of 300 s; 1/1 gates passed; One wave of 16 concurrent public requests stood for the suite's 3,153 graded ones at that concurrency; MATH-500 and MMLU-Pro are held out.
9 gates
- ✓review prerequisites
- ✓ternary fraction · 3,569,090,560 of 3,625,819,136 audited parameters (text-model weights outside the input embedding and output head, and their scales) are ternary codes: 0.984354, which must exceed 0.95
- ✓stored parameters · 3,625,819,136 stored (codes, scales, retained floats and 0 in tensors outside the manifest, uint8/int8 counting four each) against 1.1 x 3,570,052,096 original parameters
- ✓weight audit · inspect_ternary passed, as verify requires before scoring
- ✓startup · GET /health answered 200 after 152.2 s
- ✓GPQA request as graded · {"finish_reason": "length", "note": "cut off at max_tokens: grading scores such a reply zero", "content_characters": 0, "content_end": ""}
- ✓slowest of 16 concurrent requests · 16 public requests at once (8 GPQA, 8 hard LiveCodeBench) as graded (max_tokens 4096, temperature 0, 300 s each): slowest 291.2 s, median 290.1 s; 16 stopped at max_tokens, 0 timed out. Grading sends every benchmark 16 at a time, scores a request over 300 s or still failing as wrong with no limit on how many; held-out requests may run longer.
- ✓concurrent requests answered · 16 well-formed replies
- ✓healthy after requests