Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $9.89 of $10
- Model cost
- $2.60
- Agent time
- 3h 40m
- Steps
- 299
- Tokens in / out
- 27.8M / 78k
- Self-checks
- 0 passed of 4 used
- Rank at $10 wallet
- 2nd of 19
Score
0.0raw metric 0.2544
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startarchitecture.json declares deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. In t
architecture.json declares deepgrove/maple-preview at the pinned revision ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07. In trajectory#136 the shipped weights/maple shards are copied straight from the pinned /artifacts/maple/weights. The shipped config.json and the nine shard inventories match the original Maple structure (256-expert MoE, hidden size 2048). No step was found that fine-tunes or replaces these weights.
✓only vision encoder and connector addedThe only new components are a Qwen2.5-VL vision transformer and its patch merger (the connector). In trajectory#116 they
The only new components are a Qwen2.5-VL vision transformer and its patch merger (the connector). In trajectory#116 they were extracted as the visual.* tensors (390 of them) from Qwen/Qwen2.5-VL-3B-Instruct into weights/vision/vision.safetensors, with vision_config.json setting out_hidden_size to 2048. serve.py feeds the image features into Maple's inputs_embeds. It loads no other language model and adds no extra heads or adapters. The only change on the Maple side is fa3.py, a drop-in SDPA attention kernel with no new parameters.
✓main network ternaryweights/maple/config.json keeps quantize: true. modeling_maple.py is the original ternary MoE implementation, and fa3.py
weights/maple/config.json keeps quantize: true. modeling_maple.py is the original ternary MoE implementation, and fa3.py swaps only the attention kernel for SDPA. The Maple shards are the unmodified pinned weights (trajectory#136), and no step re-quantizes them or replaces them with non-ternary weights.
✓training data provenanceTRAINING_EVIDENCE.md and serve.py show no training, calibration or selection: the vision tower is used zero-shot with it
TRAINING_EVIDENCE.md and serve.py show no training, calibration or selection: the vision tower is used zero-shot with its pretrained Qwen2.5-VL weights. During the session the participant only explored the public COCO-karpathy train split (trajectory#227) and checked behaviour on the allowed /task/public examples (trajectory#234). No MMMU beyond the public examples, no held-out data, nothing from the frontier-ml-tasks repo and no validation/test splits were used for fitting. Since nothing was trained, no undeclared corpus is involved.
✓actual inference uses reviewed weightsserve.sh installs from the local wheels and then runs serve.py. serve.py loads Maple from weights/maple with AutoModelFo
serve.sh installs from the local wheels and then runs serve.py. serve.py loads Maple from weights/maple with AutoModelForCausalLM.from_pretrained(trust_remote_code=True), which runs the shipped modeling_maple.py and fa3.py. It builds Qwen2_5_VisionTransformerPretrainedModel from weights/vision/vision_config.json and calls load_state_dict on weights/vision/vision.safetensors. Every answer, whether a multiple-choice letter from logits or greedy generation, comes from model(inputs_embeds=...) or model.generate(inputs_embeds=...) on these reviewed weights, and serve.py makes no external API calls.
✓harness credentials used only by the harnessIn the trajectory, HF_TOKEN is used only to download Qwen/Qwen2.5-VL-3B-Instruct with snapshot_download (trajectory#110)
In the trajectory, HF_TOKEN is used only to download Qwen/Qwen2.5-VL-3B-Instruct with snapshot_download (trajectory#110), and Spin CLI/S3 only for compute and file transfer (trajectory#138). Both are allowed. Searches found no use of proxy or API credentials outside the harness, and the shipped serve.py contains no credentials or external endpoints.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed2h 1m into the runtook 22m 20s
startup 107.0 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 630.2 s, their 598.8 s share of the deadline after a 7.1 s first question, so the measurement is a lower bound; gradin
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 107.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 630.2 s, their 598.8 s share of the deadline after a 7.1 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
- 2failed2h 30m into the runtook 22m 18s
startup 111.8 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 650.1 s, their 599.0 s share of the deadline after a 5.8 s first question, so the measurement is a lower bound; gradin
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 111.8 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 650.1 s, their 599.0 s share of the deadline after a 5.8 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
- 3failed3h 0m into the runtook 20m 30s
startup 104.0 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 611.8 s, their 599.0 s share of the deadline after a 6.1 s first question, so the measurement is a lower bound; gradin
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 104.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 611.8 s, their 599.0 s share of the deadline after a 6.1 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
- 4failed3h 20m into the runtook 25m 55s
startup 108.4 s of 300 s; 0/1 gates passed; failed evaluation deadline: projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 789.9 s, their 599.2 s share of the deadline after a 4.7 s first question, so the measurement is a lower bound; gradin
4 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 108.4 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "A"}
- ✗evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions had not finished after 789.9 s, their 599.2 s share of the deadline after a 4.7 s first question, so the measurement is a lower bound; grading allows 3600 s for the 900 held-out questions
The agent's last message before its session ended.
Submitted.