Inverse dynamics model
- Score
- 48.1
- Wallet spend
- $33.82 of $50
- Model cost
- $31.02
- Agent time
- 10h 59m
- Steps
- 377
- Tokens in / out
- 87.2M / 259k
- Self-checks
- 2 passed of 2 used
- Rank at $50 wallet
- 7th of 14
Score
48.1raw metric 0.4807
The score is the raw metric (micro action F1), which already runs from 0 to 1, times 100.
Verifier details
- metric
- micro_action_f1
- tp
- 462
- fp
- 541
- fn
- 457
- precision
- 0.4606
- recall
- 0.5027
- failed_clip_limit_percent
- 5
- dropped_actions
- 0
- elapsed_seconds
- 833.8912
- deadline_seconds
- 3600
- request_timeout_seconds
- 300
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenancetrain.py reads only the participant's synthetic clip directories (/root/syn, /root/syn2), which training/gen.py generate
train.py reads only the participant's synthetic clip directories (/root/syn, /root/syn2), which training/gen.py generates procedurally with PIL and system fonts. It also reads the one provided example clip and its labels from /task/public/validation.jsonl, repeated 12 times, which the policy allows. Model selection used a held-out portion of the participant's own synthetic data plus the provided example. DATA_MANIFEST.md and training_log.txt state this provenance. In the trajectory, the data transfers move only the participant's own synthetic set, real_labels.json and clip.mp4 through their Spin bucket (trajectory#141). I searched for idm-eval, load_dataset, hf_hub_download and other dataset downloads. The only hits were the task prompt and the manifest's exclusion statements; there is no access to p-doom/idm-eval and no use of a public validation or test split.
✓non specialist initializationThe declared base is Qwen/Qwen2.5-VL-7B-Instruct, a general vision-language model. train.py loads that base from the hub
The declared base is Qwen/Qwen2.5-VL-7B-Instruct, a general vision-language model. train.py loads that base from the hub by name and attaches a fresh LoRA adapter; round 2 continued from the participant's own round-1 adapter. merge.py merges the round-1 checkpoint-325 adapter into the same base. The trajectory shows only snapshot_download of Qwen/Qwen2.5-VL-7B-Instruct, the merge of the participant's own ckpt325 adapter (trajectory#263), and a copy of that merged model into the output directory (trajectory#293). The config.json and weight inventory match the standard Qwen2.5-VL-7B layout (28 layers, hidden size 3584, vocabulary of 152064). Nothing shows a start from p-doom/idm or any other specialist IDM checkpoint.
✓actual inference uses reviewed weightsserve.sh installs vendored wheels offline and runs server.py. server.py loads the shipped model/ directory with Qwen2_5_
serve.sh installs vendored wheels offline and runs server.py. server.py loads the shipped model/ directory with Qwen2_5_VLForConditionalGeneration. For each request it decodes the uploaded MP4, samples 25 frames, runs generation, parses the output into actions and merges in classical computer-vision move and scroll detections from cv_detect.py, which work on the received frames. The clip id is ignored, and there is no remote call, hash lookup or stored label table. The flash_attn wheel came from the official Dao-AILab GitHub release and is used only as an attention kernel. Its contents were not reviewed, but nothing shows it producing the outputs.
✓harness credentials used only by the harnessSearches of the trajectory for API_KEY, BASE_URL, curl, claude, printenv and model-API use found no participant command
Searches of the trajectory for API_KEY, BASE_URL, curl, claude, printenv and model-API use found no participant command that used, echoed or copied the harness credentials, and no additional harness sessions were launched. The only OpenRouter-related command was `spin openrouter --help`, and API_KEY/BASE_URL matched only Spin documentation text. Network use was limited to Spin compute and storage, Hugging Face model downloads and the flash-attn release wheel from GitHub.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed7h 28m into the runtook 8m 6s
startup 71.7 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 71.7 s
- ✓POST /actions · {"actions": 16, "dropped_actions": 0, "warning": null, "first": [{"frame": "F00", "type": "MouseMove", "details": "-1,-1"}, {"frame": "F01", "type": "MouseMove", "details": "-10,-10"}, {"frame": "F03", "type": "MouseClick", "details": "Left"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 13.71 s, the first (cold) 17.09 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 110.1 s on the server the smoke request warmed up, 591.7 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 3.4 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
- 2passed10h 48m into the runtook 9m 8s
startup 87.6 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 87.6 s
- ✓POST /actions · {"actions": 16, "dropped_actions": 0, "warning": null, "first": [{"frame": "F00", "type": "MouseMove", "details": "-1,-1"}, {"frame": "F01", "type": "MouseMove", "details": "-10,-10"}, {"frame": "F03", "type": "MouseClick", "details": "Left"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 13.29 s, the first (cold) 17.42 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 105.0 s on the server the smoke request warmed up, 564.6 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 4.2 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 05:05 UTC. Summary of the delivered solution:
Approach — Hybrid inverse-dynamics model in /workspace/output/ (74 files, 19.9 GB):
- VLM: Qwen2.5-VL-7B-Instruct (general pretrained base, rev
cc594898…, not an IDM specialist) LoRA fine-tuned (r=32, language-model layers only) to emit per-frameKeyPress/MouseClick/MouseScroll/MouseMoveJSON from 25 frames at 5 FPS. LoRA merged into bf16 weights undermodel/. - Classical CV (cv_detect.py): cursor-blob tracking at native FPS for
MouseMoveand phase-correlation forMouseScroll, union-merged with VLM output in server.py.
Training data (full provenance in DATA_MANIFEST.md) — no p-doom/idm-eval clip/label was downloaded, viewed, or trained on except the one provided example:
- 5,384 self-generated synthetic screen recordings (PIL-rendered macOS-style apps with scripted actions, gen.py), plus the provided example clip upsampled ×12 (1,711-clip round-1 run; loss 0.53→0.26, checkpoint-325 packaged).
- A second round (2,003 prefilled-text clips + replay) scored higher on synthetic held-out clips (0.847–0.877 vs 0.833 F1) but was rejected: on the real example clip it hallucinated typing from static on-screen text (19–35 false
KeyPress, real F1 0.25–0.30 vs round-1's 0.47–0.60). Choice documented in MODEL_CHOICE.txt.
Measured dev quality (60 held-out synthetic clips + example clip): synthetic micro-F1 0.833 (Move 0.93, Scroll 0.72, Click 0.64, Key 0.61); example clip 0.60 with CV union.
Verification: offline wheel install + server endpoints smoke-tested locally; unit tests pass (tests/test_server.py, 3 passed); frontier selfcheck passed twice on the grading infrastructure — final run: startup 87.6s of 300s, 3/3 gates, ~14s/clip, 0 failed clips.