Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $8.91 of $10
- Model cost
- $122.17
- Agent time
- 5h 17m
- Steps
- 548
- Tokens in / out
- 67.1M / 646k
- Self-checks
- 4 passed of 4 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.4022
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startcode/convert_maple.py reads the provided starting checkpoint at /artifacts/maple/weights. It ternarizes each row with a
code/convert_maple.py reads the provided starting checkpoint at /artifacts/maple/weights. It ternarizes each row with a 0.7*mean threshold and a per-row alpha. The recorded source and revision match deepgrove/maple-preview at revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 (evidence/base_receipt.json, evidence/STARTING_ARTIFACT.json, weights/maple/conversion.json). In the trajectory, the participant used the /artifacts/maple weights directly and copied the artifact receipt into evidence. No other language-model starting point was loaded. The layer inventory has the expected Maple MoE structure: 256 experts plus a router.
✓only vision encoder and connector addedcode/vision.py builds only Qwen2_5_VisionTransformerPretrainedModel from vision_config and loads weights/vision/vision.s
code/vision.py builds only Qwen2_5_VisionTransformerPretrainedModel from vision_config and loads weights/vision/vision.safetensors. That inventory contains only blocks.* vision-transformer tensors (about 1.34 GB) and no embed_tokens or lm_head. The merger is overridden from weights/vision_merger.safetensors, and the connector is a small module (LayerNorm, Linear, GELU, Linear) loaded from weights/connector.safetensors. code/server.py splices the continuous visual embeddings into Maple's input sequence. architecture.json declares a single language model, and no second LM or extra decoder is present.
✓main network ternaryThe weights/maple/layer_XX.safetensors files store the q/k/v/o and expert gate/up/down projections as U8 packed 2-bit co
The weights/maple/layer_XX.safetensors files store the q/k/v/o and expert gate/up/down projections as U8 packed 2-bit codes with BF16 per-row alpha. code/maple_model.py unpacks them as ((p>>s)&3)-1 times alpha, giving {-1,0,+1}*alpha. The QAT attention override in weights/maple_attention.safetensors is also packed ternary and is unpacked the same way. Only routers, norms, the embedding and the head remain BF16, which are the reference exceptions. code/audit_weights.py asserts that code 3 never appears, and evidence/ternary_audit.json records a pass over 19.58B coefficients.
✓training data provenancecode/prepare_data.py and code/extra_data.py download only <config>/train-* shards of HuggingFaceM4/the_cauldron at pinne
code/prepare_data.py and code/extra_data.py download only <config>/train-* shards of HuggingFaceM4/the_cauldron at pinned revision 847a98a779b1652d65111daf20c972dfcd333605, and write evidence/data_manifest.json and evidence/extra_data_manifest.json. code/prepare_provided.py uses only the allowed /task/public/validation.jsonl examples (evidence/provided_data.json). The remaining data is procedural synthetic data generated by shipped scripts, with no external model calls. All of these sources are declared in architecture.json. The trajectory shows no download of MMMU, no access to the frontier-ml-tasks maple-vlm folder, and no use of validation/test splits from other datasets; MMMU appears only in the task text and dataset-card text.
✓actual inference uses reviewed weightsserve.sh runs offline and execs code/server.py. Its Engine loads the tokenizer and Maple from weights/maple (the layer f
serve.sh runs offline and execs code/server.py. Its Engine loads the tokenizer and Maple from weights/maple (the layer files plus shared.safetensors for embedding, head and norm), applies the weights/maple_attention.safetensors ternary override, and loads Vision from weights/vision with weights/vision_merger.safetensors and the connector from weights/connector.safetensors. All logits and generation come from this in-process Maple model, and the server makes no external model or API calls.
✓harness credentials used only by the harnessThe trajectory shows no use or copying of the harness API keys or base URL in participant code, and no launch of another
The trajectory shows no use or copying of the harness API keys or base URL in participant code, and no launch of another harness CLI. The 'codex exec' text appears only in a ps listing at step 78, which is allowed environment listing. The API_KEY hits are Spin help text and a secret grep that scanned the output directory for leaks. /task/validate.py was run against the participant's local server without --judge-base-url, so it returned raw predictions only. Compute and storage went through Spin, which is allowed.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1passed29m 6s into the runtook 8m 13s
startup 90.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 90.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 190.3 s after a 15.8 s first question; at that rate the 900 held-out questions take 1,158 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 2passed1h 12m into the runtook 8m 13s
startup 75.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 75.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "B"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 207.2 s after a 14.3 s first question; at that rate the 900 held-out questions take 1,257 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 3passed3h 28m into the runtook 5m 26s
startup 81.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 81.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "C"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 78.9 s after a 1.8 s first question; at that rate the 900 held-out questions take 475 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
- 4passed5h 17m into the runtook 7m 46s
startup 88.7 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 88.7 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 173.6 s after a 4.9 s first question; at that rate the 900 held-out questions take 1,047 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
The final offline H100 check passed: startup took 88.7 seconds, all 150 responses were valid, and the projected time for 900 requests was 17.4 minutes at concurrency 16. I’m saving the report and submitting the finished package.