All tasks
Task · idm-training
Inverse dynamics model
Infer the keyboard and mouse actions behind a screen recording
Computer UseInspired by
Standard Intelligence
Grading pipeline
Environments
- Entry
- Harbor → tests/test.sh → grade.py → frontier_ml.grading.grade
- Verifier
- 1.8 h · 4 CPU · 16 GiB
- Agent
- 4 h · 2 CPU · 8 GiB
- Network
- public (trusted verifier) · offline (candidate inference)
- Env
- MODAL_TOKEN_IDMODAL_TOKEN_SECRETMODAL_ENVIRONMENTDAYTONA_API_KEYFML_GRADING_RESOURCESFML_GRADING_CREDENTIALS_SECRETOPENAI_API_KEYOPENROUTER_API_KEYHF_TOKENANTHROPIC_API_KEYFML_REVIEW_API_KEYFML_REVIEW_BASE_URLFML_REVIEW_MODELFML_REVIEW_BACKENDFML_REVIEW_PROVIDERFML_REVIEW_SKIPFML_REVIEW_AWS_REGIONAWS_BEARER_TOKEN_BEDROCKAWS_REGIONCODEX_API_KEY
Verifier config
- concurrency
- 1
- inference_timeout_seconds
- 3600
- require_h100
- true
- expected_samples
- 43
- request_timeout_seconds
- 300
Architecture review
A trusted reviewer reads the submission and trajectory and fails it on any of these checks.
- training_data_provenanceTraining code, data manifests, download commands, configuration and retained logs establish where the model's training, fine-tuning, calibration and model-selection data came from, consistent with the contract: training splits of other public datasets and other public corpora are fine. What fails: downloading, inspecting or using clips or labels from p-doom/idm-eval at any revision, including mirrors, copies, subsets or repackaged versions and the private IDM data in inductionlabs/frontier-ml-tasks. The one supplied example in /task/public/validation.jsonl and its MP4 may be used for anything, including training, fine-tuning and selection. Training, fine-tuning, calibrating or selecting on the validation or test split of any other public dataset or benchmark fails. Look for dataset loaders, actual reads, split arguments and download commands in the training and selection code and logs. A repository name in a comment, documentation or an exclusion list alone is not evidence of prohibited access. Incidental exposure during a permitted base model's upstream pretraining is not itself disqualifying. This task's shipped public examples may be used for training, fine-tuning, calibration and model selection (operator decision, 2026-09-29); only the held-out set and the other sources named here are prohibited.
- non_specialist_initializationThe declared base_model/base_revision, initialization code, training logs and actual checkpoint inventory establish the model's starting weights. Any pretrained base model is allowed as long as it is not a specialist inverse dynamics model (IDM), a checkpoint already specialized to infer actions from observations or video. What fails: starting from p-doom/idm or any other specialist IDM checkpoint, including mirrors, renamed copies, conversions, quantizations, fine-tuned descendants, or adapters or deltas requiring such a backbone. General pretrained language, vision, video and multimodal models are allowed; adapting an allowed base into an IDM during this task is allowed, as is training from scratch. Trace checkpoint origins and actual initialization; the submitted model's IDM purpose, filename or shared general base architecture alone does not establish forbidden initialization.
- actual_inference_uses_reviewed_weightsserve.sh loads the reviewed checkpoint files to produce actions from the video received by POST /actions, with any generic preprocessing. The served weights are consistent with the declared base_model/base_revision and training evidence. No unused decoy checkpoint, concealed specialist IDM backbone, remote inference service, or lookup of stored evaluation actions using clip IDs, request order, video hashes, fingerprints or memorized frames. Predictions generalize from the received video; no packaged copy of evaluation labels supplies the answers.