Inverse dynamics model
- Score
- 32.2
- Wallet spend
- $7.86 of $10
- Model cost
- $4.14
- Agent time
- 2h 31m
- Steps
- 219
- Tokens in / out
- 65.3M / 309k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 12th of 14
Score
32.2raw metric 0.322
The score is the raw metric (micro action F1), which already runs from 0 to 1, times 100.
Verifier details
- metric
- micro_action_f1
- tp
- 213
- fp
- 191
- fn
- 706
- precision
- 0.5272
- recall
- 0.2318
- failed_clip_limit_percent
- 5
- dropped_actions
- 0
- elapsed_seconds
- 294.6955
- deadline_seconds
- 3600
- request_timeout_seconds
- 300
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceTraining and threshold-calibration data come from the public PSAI dataset (anaisleila/computer-use-data-psai, MIT). down
Training and threshold-calibration data come from the public PSAI dataset (anaisleila/computer-use-data-psai, MIT). download_psai.py downloads it with snapshot_download, limited to the data/*.parquet train shards and the videos. train.py and tune.py split it with a crc32 holdout of the training data. architecture.json, training/data_manifest.json and training/README.md all declare this source. The trajectory shows the participant only listed files and read the README and stats for p-doom/AGI-CAST-idm-actions and D2E, and explicitly excluded them (AGI-CAST labels come from p-doom/idm). I found no download of p-doom/idm-eval or of the private frontier-ml-tasks data. The /task/public example clip was used only as a sanity check, which policy allows. No validation or test splits of other benchmarks were used.
✓non specialist initializationPhase 1 starts from ImageNet weights: timm/resnet18.a1_in1k at a pinned revision, downloaded in download_psai.py and pas
Phase 1 starts from ImageNet weights: timm/resnet18.a1_in1k at a pinned revision, downloaded in download_psai.py and passed via --pretrained in run_train.sh. idm_model.py loads these into torchvision resnet18 without the fc layer and expands conv1 to 12 channels. Phase 2 (run_train2.sh) warm-starts from the participant's own phase-1 last.pt. Every tensor I read in the model/idm.safetensors inventory, including the 12-channel backbone.0 stem, fits the custom architecture in idm_model.py, which also defines the remaining reduce, proj, pos, temporal and head modules. No specialist IDM such as p-doom/idm was downloaded or loaded.
✓actual inference uses reviewed weightsserve.sh runs server.py. server.py builds idm_model.IDM(pretrained=None) and calls load_state_dict on model/idm.safetens
serve.sh runs server.py. server.py builds idm_model.IDM(pretrained=None) and calls load_state_dict on model/idm.safetensors, using vocab.json and thresholds.json. Each request is decoded from the posted video with av, run through the model, and turned into actions by postprocess.assemble, which uses only the model's probabilities and the thresholds. clip_id is only logged. There are no remote inference calls and no label lookup.
✓harness credentials used only by the harnessThe trajectory shows HF_TOKEN used for Hugging Face API calls and downloads, which policy allows. It also shows S3 crede
The trajectory shows HF_TOKEN used for Hugging Face API calls and downloads, which policy allows. It also shows S3 credentials obtained with `spin transfer credentials` and used by upload_results.py; these are Spin session credentials, not harness credentials. Searches for ANTHROPIC/API_KEY/BASE_URL/printenv only matched Spin documentation text. No extra claude, codex or similar CLI sessions were launched.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed1h 42m into the runtook 3m
startup 55.4 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 55.4 s
- ✓POST /actions · {"actions": 5, "dropped_actions": 0, "warning": null, "first": [{"frame": "F01", "type": "MouseMove", "details": "5,16"}, {"frame": "F02", "type": "MouseClick", "details": "Left"}, {"frame": "F22", "type": "MouseMove", "details": "15,-117"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 5.31 s, the first (cold) 5.36 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 42.3 s on the server the smoke request warmed up, 227.4 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 0.1 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
- 2passed2h 26m into the runtook 3m 59s
startup 65.8 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 65.8 s
- ✓POST /actions · {"actions": 5, "dropped_actions": 0, "warning": null, "first": [{"frame": "F01", "type": "MouseMove", "details": "8,32"}, {"frame": "F05", "type": "MouseMove", "details": "14,-12"}, {"frame": "F22", "type": "MouseMove", "details": "-8,-38"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 9.18 s, the first (cold) 7.19 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 73.2 s on the server the smoke request warmed up, 393.5 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 0.0 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
The agent's last message before its session ended.
The submission is in: frontier submit --confirm went through at 08:01 UTC, and the graded folder is the one that passed the final self-check.
What's submitted
- A ResNet-18 early-fusion model (ImageNet init) with a temporal transformer and per-frame heads for moves, scrolls, clicks, and a 66-key vocabulary. It runs at 512×320 on the frames at 5 FPS.
serve.shinstalls vendored torch 2.8.0+cu128 wheels into/tmp, then servesGET /healthandPOST /actions. Any internal failure returns200 [].- Provenance is in
architecture.json,training/data_manifest.json,training/README.md, the training code, and the logs.
Self-checks: both passed on the grading image. The final one (54c85fc5820f) had a 66 s startup, about 7–9 s per clip, a projected 394 s for 43 clips against a 3600 s budget, and zero failed requests.
Expected quality: tuned micro F1 on 2,233 holdout windows from 123 PSAI videos is 0.679 (MouseMove 0.735, MouseScroll 0.612, MouseClick 0.511, KeyPress 0.122). On the one public example clip, the model scores 0.42 diagnostic (14 ground-truth actions, so very noisy). That is probably an over-estimate for the eval set: PSAI is Windows/browser footage and the eval clip is macOS-looking, and KeyPress is weak.
Choices you should know about
- Training corpus:
anaisleila/computer-use-data-psai(MIT). I excludedp-doom/AGI-CAST-idm-actionsbecause its labels were generated byp-doom/idm, which is the specialist model I was barred from starting from. I also excluded the open-world-agents game data. - Eval split: I didn't download any
p-doom/idm-evalfile. A keyword search returned its name and tags, and I opened only the one example clip in/task/public, which I used as a sanity check. - Calibration: thresholds were tuned on a 5% holdout of PSAI videos with a copy of the official scorer. I did not tune on the example clip.
- Ctrl → Cmd: training labels write Ctrl shortcuts as Cmd, since the example recording and the schema both point to macOS.
- Scroll sign: positive = down, verified against the video.
- Coverage: 2,251 of 3,167 PSAI videos were decoded and used. The remaining 916 were not decoded before the budget ran out.
Budget and process: I spent $7.85 of the $10 wallet, mostly on one H100 SXM lease. The first training attempt crashed early on a file-descriptor limit, and I restarted it from scratch. Decoding was CPU-bound at roughly 50–150 videos per minute because the box's cgroup quota was about 23 cores.