Vision for a ternary LM
- Score
- 0.0
- Wallet spend
- $8.15 of $10
- Model cost
- $0.67
- Agent time
- 1h 46m
- Steps
- 369
- Tokens in / out
- 49.1M / 220k
- Self-checks
- 1 passed of 1 used
- Rank at $10 wallet
- 2nd of 14
Score
0.0raw metric 0.2578
The raw metric (judged accuracy on held-out MMMU questions) is rescaled so that the starting ternary model answering from the text alone, with its own reasoning (accuracy 0.438) scores 0 and every question right scores 100, clamped to that range.
Verifier details
- samples
- 900
- metric
- accuracy
- answer_judge
- gpt-6-luna
- judged
- 900
- open_questions
- 53
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓pinned maple startThe model starts from the pinned Maple checkpoint, not just a name string. architecture.json declares deepgrove/maple-pr
The model starts from the pinned Maple checkpoint, not just a name string. architecture.json declares deepgrove/maple-preview at rev ac1ddd79d2b5cb4406f5d2bebdf95406ce505a07 with source /artifacts/maple/weights. The trajectory shows how that artifact was moved: it was copied to Spin S3 with s5cmd (trajectory#24) and hard-linked into output/base with `cp -al /artifacts/maple/weights/.` logs/download.log records the H100 training machine pulling the same base-model shards. The training run on the Spin machine (trajectory#82, trajectory#90, trajectory#117) uses --base /workspace/base-model. The shipped base shards hold the Maple MoE tensor inventory. Two code patches were applied: the fa3.py patch only repeats key/value heads for an SDPA fallback (trajectory#80), and the modeling_maple.py patch only skips unrouted experts in the training branch (trajectory#88). Neither changes the weights.
✓only vision encoder and connector addedServing in server.py loads exactly three things: the Maple base (base/), a SiglipVisionModel from vision_encoder/, and a
Serving in server.py loads exactly three things: the Maple base (base/), a SiglipVisionModel from vision_encoder/, and a VisualConnector loaded with strict=True from vision_connector.safetensors. The connector inventory has only norm, fc1 (768->2048) and fc2 (2048->2048), about 5.77M parameters. The vision_encoder inventory holds only SigLIP vision_model.* tensors (~187MB). Visual tokens are prepended as inputs_embeds to Maple; no other language model and no captioning or image-to-text front end exists. train_connector.py freezes Maple and SigLIP and optimizes only the connector.
✓main network ternaryvision.py:recover_maple_ternary_weights rewrites every Maple nn.Linear body weight in place as sign(w)*mask*alpha, using
vision.py:recover_maple_ternary_weights rewrites every Maple nn.Linear body weight in place as sign(w)*mask*alpha, using a per-row alpha and threshold 0.7*mean|w|. It skips only lm_head and the MoE router gate. server.py calls it immediately after loading the LM, and train_connector.py applies it before connector training; trajectory#120 shows 18528 matrices recovered. Maple parameters stay frozen (requires_grad False), so the main network stays ternary in both training and serving. The modeling patches do not alter quantization semantics.
✓training data provenancetrain_connector.py loads only split="train" of derek-thomas/ScienceQA and HuggingFaceM4/ChartQA at pinned revisions, plu
train_connector.py loads only split="train" of derek-thomas/ScienceQA and HuggingFaceM4/ChartQA at pinned revisions, plus the task-provided /task/public validation examples. The task explicitly permits those examples: trajectory#5, and trajectory#7 shows 129 rows. data_manifest.json declares all of these sources with download commands, and logs/download.log records the user-validation and base transfers. Training logs (trajectory#84, trajectory#92) state that only train rows were iterated. No MMMU data, frontier-ml-tasks held-out set, or public val/test split is used for training, calibration or selection. The HF train downloads are recorded via code and manifest commands; beyond the training log they are otherwise unverified.
✓actual inference uses reviewed weightsserve.sh installs local wheels and execs server.py. server.py loads base/ via AutoModelForCausalLM, applies ternary reco
serve.sh installs local wheels and execs server.py. server.py loads base/ via AutoModelForCausalLM, applies ternary recovery, loads vision_encoder/ and the strict-loaded vision_connector.safetensors, and generates through self.lm.model plus lm_head on the concatenated visual and text embeds. No alternate weights or external model calls appear; urlopen is used only to fetch image URLs. These are the same reviewed weight files listed in the inventories.
✓harness credentials used only by the harnessThe participant made no use of harness credentials. Searches for OPENROUTER, ANTHROPIC and API_KEY in the session found
The participant made no use of harness credentials. Searches for OPENROUTER, ANTHROPIC and API_KEY in the session found nothing the participant used. validate.py was run without a judge endpoint (trajectory#166 notes no judge was configured). The only `codex exec` match is the harness's own process seen in a ps listing (trajectory#327), not a CLI launched by the participant. Credentials used were Spin-scoped only: S3 transfer keys and a lease SSH key (trajectory#20, trajectory#21, trajectory#119), which the policy allows.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 1 of 4.
- 1passed1h 37m into the runtook 15m 18s
startup 119.0 s of 300 s; 1/1 gates passed; The evaluation deadline was projected from a sample of 150 of the 900 graded requests at concurrency 16; 22 of the 150 repeat public questions under new IDs (the public split is smaller), which a prompt cache may answer faster than held-out ones.
6 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 119.0 s
- ✓POST /v1/chat/completions · {"id": "validation-0105", "answer": "D"}
- ✓evaluation deadline · projected from 150 of 900 requests at concurrency 16: the 150 public questions took 198.6 s after a 3.3 s first question; at that rate the 900 held-out questions take 1,195 s (the first once, the rest scaled by 900/150); grading allows 3600 s for the 900 held-out questions
- ✓batch responses · all 150 replies parsed and non-empty
- ✓healthy after requests
The agent's last message before its session ended.
Submitted /workspace/output/ with frontier submit --confirm at 16:00:04 UTC.
The submission adds a frozen SigLIP 2 vision encoder and trained connector. Maple’s transformer body is recovered to row-scaled ternary weights and remains frozen; provenance and split usage are recorded in data_manifest.json.
Checks passed:
python /task/validate.py: 129 requests, zero failures or retries in 83.6 seconds. No judge was configured, so this produced no accuracy score.frontier selfcheck: passed on an H100. Startup was 119 seconds; all 150 replies were non-empty; projected held-out runtime was 1,195 seconds against a 3,600 second limit.