Inverse dynamics model
- Score
- 34.5
- Wallet spend
- $1.13 of $10
- Model cost
- $29.05
- Agent time
- 8h 24m
- Steps
- 359
- Tokens in / out
- 83.0M / 280k
- Self-checks
- 2 passed of 2 used
- Rank at $10 wallet
- 10th of 14
Score
34.5raw metric 0.3454
The score is the raw metric (micro action F1), which already runs from 0 to 1, times 100.
Verifier details
- metric
- micro_action_f1
- tp
- 291
- fp
- 475
- fn
- 628
- precision
- 0.3799
- recall
- 0.3166
- failed_clip_limit_percent
- 5
- dropped_actions
- 0
- elapsed_seconds
- 235.5206
- deadline_seconds
- 3600
- request_timeout_seconds
- 300
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceAll training, validation and threshold-tuning data came from the participant's own synthetic recordings. These were made
All training, validation and threshold-tuning data came from the participant's own synthetic recordings. These were made on a Spin VM by driving Firefox under Xvfb with XTEST input, using procedurally generated web apps (pages.py/record2.py) and scripted browsing of public no-login websites (record_web.py). The sources are declared in architecture.json and TRAINING.md. Labels came from cut2.py and make_tune.py, which read the participant's own log.json/video.mp4 sessions and apply copies of the evaluator's aggregation functions. train.py, train_keycrop.py and tune_run.py split by session within this self-generated data. The only public clip was the supplied validation example, which the policy allows. Searching the trajectory for p-doom/idm-eval, huggingface, hf_hub and load_dataset found only the task prompt and the participant's documentation statements. No download of a forbidden dataset or of the private frontier-ml-tasks data appears, and no public val/test split was used. The live-website browsing corpus is declared, but its collection is recorded only by the scripts and Spin runs; no external download log exists. That is noted as unverified but acceptable.
✓non specialist initializationtraining/train.py builds resnet18(weights=ResNet18_Weights.IMAGENET1K_V1) and widens conv1 to 12 channels, adding heads
training/train.py builds resnet18(weights=ResNet18_Weights.IMAGENET1K_V1) and widens conv1 to 12 channels, adding heads for move, scroll, click, keys, mods and cursor. training/train_keycrop.py uses the same ImageNet ResNet-18 with fc resized to 47 classes. The model.onnx tensor inventory (conv1 [64,12,7,7], a ResNet-18 trunk and these exact heads) and the keycrop.onnx inventory (ResNet-18 with fc [47,512]) match those scripts. architecture.json declares the torchvision IMAGENET1K_V1 base. No load of p-doom/idm or any other specialist IDM checkpoint appears in the code or the trajectory.
✓actual inference uses reviewed weightsserve.sh installs local wheels offline and runs exec python server.py. server.py loads only the local model.onnx and key
serve.sh installs local wheels offline and runs exec python server.py. server.py loads only the local model.onnx and keycrop.onnx through onnxruntime. Each request decodes the uploaded video, samples frames, builds frame-triplet and diff inputs, runs both ONNX models and fuses their outputs with OpenCV classical scroll and change analysis. The request id is ignored, and the server makes no network calls. thresholds.json holds only thresholds and label vocabularies, not per-clip answers, so there is no label lookup. The inventories show that both served ONNX files are the reviewed ResNet-18 models, with no concealed IDM backbone or remote inference service.
✓harness credentials used only by the harnessThe trajectory shows no use of harness API keys or base URLs in participant code. Searches for API_KEY, BASE_URL, anthro
The trajectory shows no use of harness API keys or base URLs in participant code. Searches for API_KEY, BASE_URL, anthropic and openrouter hit only agent metadata and the Spin help text. The step-2 environment listing was masked. No extra harness agent CLI was launched. The participant used only Spin compute commands (allowed), standard pip and ffmpeg tooling, and the benchmark's own frontier selfcheck/submit commands.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 2 of 4.
- 1passed8h 17m into the runtook 1m 18s
startup 6.2 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 6.2 s
- ✓POST /actions · {"actions": 8, "dropped_actions": 0, "warning": null, "first": [{"frame": "F04", "type": "MouseMove", "details": "-2,-1"}, {"frame": "F05", "type": "MouseMove", "details": "-3,0"}, {"frame": "F06", "type": "MouseMove", "details": "-3,0"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 3.82 s, the first (cold) 4.10 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 31.2 s on the server the smoke request warmed up, 167.6 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 0.3 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
- 2passed8h 20m into the runtook 1m 31s
startup 4.9 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 4.9 s
- ✓POST /actions · {"actions": 8, "dropped_actions": 0, "warning": null, "first": [{"frame": "F04", "type": "MouseMove", "details": "-2,-1"}, {"frame": "F05", "type": "MouseMove", "details": "-3,0"}, {"frame": "F06", "type": "MouseMove", "details": "-3,0"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 6.13 s, the first (cold) 3.33 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 49.4 s on the server the smoke request warmed up, 265.6 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 0.0 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted successfully at 2026-10-13 02:29:40 UTC.
What was built — an inverse-dynamics server in /workspace/output/ (37 files, 192.6 MB):
- Training data (self-generated, no idm-eval data used): 72 desktop recording sessions made on a rented VM — procedurally generated web apps plus scripted browsing of real public websites (Wikipedia, GitHub, news, docs, search) in Firefox under Xvfb. All input was injected as real X11 XTEST events and captured with ffmpeg wallclock-timestamped x11grab, so video↔label alignment is exact; labels were computed with the evaluator's own aggregation semantics. Two hard-won fixes mattered: Playwright's synthetic input never moves the OS cursor (switched to XTEST), and CFR re-gridding destroys timing (switched to wallclock PTS).
- Models (both ResNet-18 from general ImageNet-1K weights — no specialist IDM init): a frame-triplet action model (move/scroll/click/key heads + cursor auxiliary) trained on 1,748 synthetic 5s clips, and a key-crop character classifier (native-res crops around text changes, 9,069 crops, ~50% top-1 over 47 classes). Exported to ONNX.
- Server (
serve.sh→ stdlib HTTP server, vendored onnxruntime/opencv wheels, offline install): fuses model output with classical screen analysis (blob-pair cursor tracking, row-profile scroll shift, change-energy gating of keys). Thresholds tuned end-to-end on 240 held-out synthetic clips (tune F1 0.57; per-type: Key 0.56, Move 0.69, Scroll 0.52; the click head was net-negative and is effectively disabled).
Verification observed: validate.py on the provided example clip scores 0.696 micro-F1 (MouseMove 0.94) at ~3.5 s/clip on CPU; two frontier selfcheck runs on the final bytes both passed all gates (startup 4.9 s of 300 s, 43-clip batch projected at ~168 s of 3600 s, 0 failed clips). Total compute spend: $1.13 of the $10 Spin budget.