Inverse dynamics model
- Score
- 46.0
- Wallet spend
- $35.18 of $200
- Model cost
- $20.95
- Agent time
- 11h 27m
- Steps
- 357
- Tokens in / out
- 97.3M / 437k
- Self-checks
- 1 passed of 4 used
- Rank at $200 wallet
- 6th of 14
Score
46.0raw metric 0.4597
The score is the raw metric (micro action F1), which already runs from 0 to 1, times 100.
Verifier details
- metric
- micro_action_f1
- tp
- 408
- fp
- 448
- fn
- 511
- precision
- 0.4766
- recall
- 0.4440
- failed_clip_limit_percent
- 5
- dropped_actions
- 0
- elapsed_seconds
- 536.1861
- deadline_seconds
- 3600
- request_timeout_seconds
- 300
After grading, a reviewer model reads the submission's files and the agent's trajectory and checks each of the task's rules. A failed check makes the score 0. Step numbers link into the trajectory.
✓training data provenanceDATA_MANIFEST.md and datagen/worker.py show where the training data came from: synthetic Playwright/Chromium clips recor
DATA_MANIFEST.md and datagen/worker.py show where the training data came from: synthetic Playwright/Chromium clips recorded on local HTML pages, plus read-only browsing of public websites listed in REAL_URLS (Wikipedia, Hacker News, arXiv and others). The only other source is the one provided example. In the session, that clip and validation.jsonl were copied from /task/public to s3://spin/zs/ and then onto the GPU machine (trajectory#40). train/mkreal.py converts that one clip, which the policy allows. Searching the session for idm-eval or p-doom turned up only the task prompt and the participant's own manifest statements. The search found no download of that dataset, no load_dataset or snapshot_download call, no wget, and no frontier-ml-tasks access. Checkpoints were selected using held-out synthetic clips and the permitted example clip.
✓non specialist initializationThe declared base model is Qwen/Qwen2.5-VL-3B-Instruct at revision 66285546..., which is a general vision-language model
The declared base model is Qwen/Qwen2.5-VL-3B-Instruct at revision 66285546..., which is a general vision-language model. logs/base_revision.txt records that revision. The session shows the model downloaded with `hf download Qwen/Qwen2.5-VL-3B-Instruct` and its revision checked through the HF API. train/sft.py applies LoRA to that base. train/merge.py merged checkpoint-1000 into the same base (trajectory#303). The shipped model/model.safetensors (7,509,338,080 bytes) is a standard Qwen2.5-VL tensor layout, and its size matches the merged output copied into the output folder. I found no specialist IDM checkpoint anywhere.
✓actual inference uses reviewed weightsserve.sh installs the vendored wheels and runs server.py with HF_HUB_OFFLINE set. server.py loads only the local model/
serve.sh installs the vendored wheels and runs server.py with HF_HUB_OFFLINE set. server.py loads only the local model/ directory (the merged Qwen2.5-VL weights). It decodes the posted MP4 into 26 frames at 5 FPS, builds prompts from the previous, current and next frames plus crops (prompt.py), and generates each frame's actions with the model. The clip id is used only in a log line. There is no remote inference call, no label lookup and no hashing or fingerprinting of videos. attn_patch.py is only an attention-speed patch.
✓harness credentials used only by the harnessThe harness credential (META_API_KEY) appeared only once, in a masked env listing (`sed 's/=.*/=<set>/'`) at trajectory#
The harness credential (META_API_KEY) appeared only once, in a masked env listing (`sed 's/=.*/=<set>/'`) at trajectory#5, and was never used or copied afterwards. HF_TOKEN was used only for read-only HF API and model-download calls, which is allowed. The Spin transfer session credentials are the participant's own, not harness credentials. Searches found no model API calls with harness credentials, no use of a harness base URL and no extra harness CLI sessions.
LLM adversarial review is not proof of arbitrary program behavior. Host storage and task-specific deterministic checks remain required.
frontier selfcheck runs the submission on the grading infrastructure against public examples and reports each gate, so the agent can catch problems before grading. Each run gets four. This agent used 4 of 4.
- 1failed10h 47m into the runtook 7m 31s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] installing vendored wheels into /tmp/idm_pkgs ... [serve] starting server ... [server] loa
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] installing vendored wheels into /tmp/idm_pkgs ... [serve] starting server ... [server] loading model from /submission/model ... Traceback (most recent call last): File "/submission/server.py", line 245, in <module> main() File "/submission/server.py", line 235, in main load_model() File "/submission/server.py", line 54, in load_model MODEL = Qwen2_5_VLForConditionalGeneration.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/modeling_utils.py", line 4192, in from_pretrained device_map = check_and_set_device_map(device_map) # warn, error and fix the device map ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/integrations/accelerate.py", line 137, in check_and_set_device_map raise ValueError( ValueError: Using a `device_map`, `tp_plan`, `torch.device` context manager or setting `torch.set_default_device(device)` requires `accelerate`. You can install it with `pip install accelerate`
- 2failed10h 56m into the runtook 7m 40s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] installing vendored wheels into /tmp/idm_pkgs ... ERROR: Could not find a version that sat
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: [serve] installing vendored wheels into /tmp/idm_pkgs ... ERROR: Could not find a version that satisfies the requirement psutil (from accelerate) (from versions: none) ERROR: No matching distribution found for psutil File "<string>", line 1, in <module> ModuleNotFoundError: No module named 'torch' [serve] starting server ... Traceback (most recent call last): File "/submission/server.py", line 251, in <module> main() File "/submission/server.py", line 241, in main load_model() File "/submission/server.py", line 40, in load_model import torch ModuleNotFoundError: No module named 'torch'
- 3failed11h 3m into the runtook 8m 27s
failed startup: Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: eights: 100%|██████████| 824/824 [00:00<00:00, 4133.69it/s] Traceback (most recent call last): F
2 gates
- ✓review prerequisites
- ✗startup · Server did not become healthy in 300s (last /health probe: RuntimeError: Submission HTTP request failed: ConnectionRefusedError: [Errno 111] Connection refused); end of its log: eights: 100%|██████████| 824/824 [00:00<00:00, 4133.69it/s] Traceback (most recent call last): File "/submission/server.py", line 251, in <module> main() File "/submission/server.py", line 241, in main load_model() File "/submission/server.py", line 69, in load_model PROCESSOR = AutoProcessor.from_pretrained(model_dir, local_files_only=True) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/models/auto/processing_auto.py", line 346, in from_pretrained return processor_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/processing_utils.py", line 1754, in from_pretrained args = cls._get_arguments_from_pretrained(pretrained_model_name_or_path, processor_dict, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/processing_utils.py", line 1894, in _get_arguments_from_pretrained sub_processor = auto_processor_class.from_pretrained( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/models/auto/video_processing_auto.py", line 433, in from_pretrained return video_processor_class.from_pretrained(pretrained_model_name_or_path, *inputs, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/tmp/idm_pkgs/transformers/utils/import_utils.py", line 2413, in __getattribute__ requires_backends(cls, cls._backends) File "/tmp/idm_pkgs/transformers/utils/import_utils.py", line 2399, in requires_backends raise ImportError("".join(failed)) ImportError: Qwen2VLVideoProcessor requires the Torchvision library but it was not found in your environment. Check out the instructions on the installatio ...[clipped]
- 4passed11h 19m into the runtook 6m 31s
startup 100.7 s of 300 s; 3/3 gates passed; Grading's 43-clip batch was sampled: 8 copies of the public clip in their 670 s share of the 3600 s deadline; its time and failed clips were projected from them.
7 gates
- ✓review prerequisites
- ✓startup · GET /health answered 200 after 100.7 s
- ✓POST /actions · {"actions": 14, "dropped_actions": 0, "warning": null, "first": [{"frame": "F00", "type": "MouseMove", "details": "-67,18"}, {"frame": "F01", "type": "MouseMove", "details": "-18,67"}, {"frame": "F03", "type": "MouseClick", "details": "Left"}]}
- ✓slowest request · 9 clips one at a time, each from its first attempt to its last (retries included), median 12.41 s, the first (cold) 14.74 s
- ✓43 clips one at a time · grading's request loop (validate.py run()) on 8 copies of the public clip: 98.9 s on the server the smoke request warmed up, 531.8 s for 43 (projected from 8 of 43 requests at concurrency 1), plus 2.3 s the cold first clip took over the median; held-out clips differ
- ✓failed clips · none of 8 (0 needed a retry)
- ✓healthy after requests
The agent's last message before its session ended.
Submitted at 2026-10-09 17:32:49 UTC.
Outcome: frontier selfcheck PASSED on the final /workspace/output (97 files, 11.56 GB), then submitted with frontier submit --confirm.
What was delivered: a fine-tuned Qwen2.5-VL-3B (LoRA merged, 7.1 GB) that infers per-frame keyboard/mouse actions from 5s screen recordings, served offline via serve.sh → POST /actions. Trained only on self-generated Playwright/Chromium synthetic clips plus the one provided example (20x upsampled); eval F1 0.786 on the real clip, 0.861/0.890 on held-out synthetic sets.
Selfcheck gates (all pass): startup 100.7s of 300s; slowest request 14.7s of 300s; 43-clip projection 534s of 3600s; 0 failed clips; valid response schema.
Last fixes this session: removed the device_map arg needing accelerate; vendored Pillow/requests; added a torchvision-free processor fallback (verified on CPU with exact image-token expansion) after three startup failures burned three selfchecks.