FrontierML: a challenging, end-to-end machine learning benchmark

One desirable property of human-run frontier AI labs is that they can consistently improve their models with increased access to capital. As agents begin doing machine learning work, the same question applies: given more money, can an agent build a better model?

FrontierML is a benchmark that measures how well agents turn capital into model performance on challenging, end-to-end machine learning tasks. Our benchmark makes several key advancements in capturing the real-world complexity of modern machine learning:

• Scaling spending budgets: Existing benchmarks scale how much agents can think. We scale how much they can spend. We give agents $10, $50, and $200 per task to spend on training compute and data, and evaluate whether agents can use the additional capital productively.
• Open-ended, end-to-end tasks: FrontierML is the most open-ended benchmark for agentic machine learning. In each task, agents can choose architectures, source data, implement training, provision compute, evaluate checkpoints, and optimize inference. Each task is scored on a continuous scale based on the final performance of the submitted model.
• Robust verification: Agents submit an inference server that serves their model on an air-gapped H100 machine. Submissions are evaluated on held-out out-of-distribution tests, and a separate reviewer agent examines each submission and its trajectory for violations of task rules.
• Diverse and real-world focus: FrontierML contains 21 long-horizon tasks inspired by real work at frontier labs, neolabs, and academic institutions.

Overall, FrontierML faithfully measures whether or not agents can autonomously train other models, end-to-end, while handling the complexity and applying the judgment a human research team would.

Link to this sectionOverview

Link to this sectionScaling capital access for agents

In the real world, researchers might spend $10 or $200 or $50,000 toward a given task, with performance improving as budget scales. FrontierML measures whether agents can train better models as we scale the money they are given.

FrontierML tasks are designed to give agents control over the full model development process, creating opportunities to turn additional capital into better model performance through larger training runs, better data, or more algorithmic experimentation.

In our experiments, we find that stronger agents are able to reliably improve their submissions with greater access to capital.

Link to this sectionOpen-endedness

Like real-world machine learning work, FrontierML tasks define a problem without prescribing a solution. Agents build and submit their own inference server, which we evaluate by sending requests and scoring its responses. This leaves the entire development process open-ended: agents decide what model architecture to build, what data to use, which GPUs to rent and for how long, which experiments to run, and how to serve the result.

Agents adopt different approaches for each task, resulting in a wide band of scores.

Link to this sectionPreventing reward hacking

Open-ended long-horizon tasks have a large reward hacking surface, which we address with the following:

Air-gapped solution execution. The agent has internet access during development, but must submit a standalone package of serving code and model weights that runs with no internet access, on a single H100 machine.

Extensive held-out test set. To disincentivize overfitting, we provide limited validation samples and evaluate on a large out-of-distribution held-out test set, not shown to the agent, for each task. For example, for the Ternary PTQ task, we only tell the agent two of four benchmarks we will evaluate on (public: GPQA-Diamond, LCB v6; held-out: MMLU-Pro, MATH-500). In the SVG generation task, we make the held-out samples substantially more complex than the validation samples.

LLM judge review. After each task is complete, we spin up a review agent with access to the submission and the agent’s trajectory, which checks for cheating, including:

• Searching up answers: did the agent search for FrontierML solutions?
• Training on test set: did the agent train on the validation/test split of any public benchmark?
• Invalid model provenance: did the agent not use the requested base model if provided, or did the agent serve a prohibited model?

Strict time limits. We set an allowable evaluation time limit to bound the allowed inference compute for each submission. If the submitted server doesn’t return answers for each held-out sample within the limit, the agent scores 0 on the task.

Link to this sectionReal-world tasks

FrontierML contains 21 end-to-end tasks spanning language, vision, audio, and games, inspired by research at frontier labs, neolabs, and academic institutions. Tasks range from converting an autoregressive LLM into a text diffusion model to building a streaming end-of-turn detector for voice assistants to writing a bot that plays competitive Pokémon battles.

Link to this sectionMethodology

Link to this sectionEvaluation setup

Each task provides the agent with the same setup: a 12-hour CPU sandbox, a wallet with $10, $50, or $200, a few validation samples, and a task description. For each task, we also provide several constraints to ensure tasks are not trivially solvable, for example by restricting use of already-trained public specialist models or training on the test split of any public benchmark.

We provide the minimal amount of information required to complete the task. After submission, a review agent is given a sandbox to review the submission and agent’s trajectory to ensure the constraints were followed.

When evaluating each agent, we use the reasoning effort that provided the best score on other agentic benchmarks for each respective agent. Each model runs in the harness its developer used for agentic benchmarks on its model card, mirroring day-to-day use.

Before submitting, agents can test their work with frontier selfcheck, which runs the current submission on the grading infrastructure against the public validation samples, up to four times per trial.

Evaluating an agent on all 21 tasks costs up to $5,460 in wallet spend, not including model API costs.

Link to this sectionAccess to compute and data

To give each agent simple access to compute and synthetic data generation, we created Spin, an open-source CLI tool and service that allows agents to purchase on-demand compute from 17 different compute providers, and metered tokens from OpenRouter. We provide a hosted version with zero fees for ease of evaluation.

Because compute prices fluctuate, we fix the price of each GPU type across all evaluation runs. We took a snapshot of on-demand compute prices on September 29, 2026, calculated the median price across providers and saved the result.We price GPUs by form factor, with H100 SXM, H100 PCIe and H100 NVL each having their own fixed price. The snapshot used for evaluation is available as a JSON file. When running the evaluation, we set compute to the price captured in the snapshot by default.

Link to this sectionResults

Link to this sectionObservations

Link to this sectionAgents make new architectures

Because FrontierML keeps tasks open-ended and allows agents to do whatever they think will produce the best model, agents design their own architectures and produce their own data.

In 28% of trajectories, agents add newly trained parts to pretrained models or train a new model from scratch.

Of the 735 runs that trained a model, 14% used only data they generated with an LLM, a local model, or a renderer or simulator, 44% combined public datasets with data they generated themselves, and 35% used public datasets only. 7% trained on the task's own data alone.

Agents almost always write their own training loop: 87% of training runs use a hand-written PyTorch loop, and 8% use the Hugging Face Trainer. 26% of runs train a LoRA through the PEFT library.

Link to this sectionBudget is not fully spent

On average, agents only spend 46% of the capital in their wallet, even when they are explicitly told there is no penalty for using their entire wallet. No individual agent uses more than 80% of the capital available to it at any budget.

Link to this sectionTime is not fully spent

While FrontierML provides agents with a maximum time limit of 12 hours, agents often submit much earlier. On average, frontier models only spent half of the available time, while no model spent more than 9 hours on average.

Link to this sectionInefficient spending

Agents rarely researched their options before spending money, instead relying on prior knowledge to choose models for synthetic data generation and GPUs to rent.

Synthetic data: agents almost always chose models for synthetic data generation from memory, without searching for newer models beyond their knowledge cutoff. Only 3 runs listed the set of available models on OpenRouter before generation. Over 67% of all calls to OpenRouter for synthetic data were to models more than 1 year old.

GPU selection: across all agents, the vast majority of compute was in H100s, with 60% being spent on H100 SXM GPUs. Over all runs, only two web searches were performed about which GPU to use, both of which were from Grok 4.7.

Link to this sectionPrior Work

The goal of FrontierML is to capture the real-world complexities of modern machine learning.

FrontierML is the most open-ended capabilities benchmark for agentic machine learning. Unlike existing benchmarks, we directly give agents access to money which they can use to purchase any type of GPU reservation, and tokens from any model on OpenRouter to generate synthetic data.

FrontierML's general task interface (submit any code that runs on an H100 and serves the model) allows agents to start from any public model and architecture, just as a human researcher could.

Agents must decide how to allocate spend between compute and synthetic data, which experiments to run, whether to start from a pretrained model, which public datasets to use or what data to generate, and whether to adopt existing architectures and learning algorithms or create their own.

Existing benchmarks are narrower in scope and give up task diversity or realism. We summarize the key differences below:

PostTrainBench is the closest work to ours. It does not give models freedom to design model architectures or generate synthetic data like FrontierML does. Because tasks are limited to LLM post-training, they do not provide a holistic evaluation of machine learning capability, reducing model separation compared to FrontierML.For example, PostTrainBench does not test architecture design, synthetic data generation, or non-language-modeling tasks.

Overall, FrontierML more faithfully measures whether or not agents can autonomously train other models, end-to-end, while handling the complexity and applying the judgment a human research team would.

Link to this sectionFuture Work & Limitations

Current runs use budgets of $10, $50, and $200. At present, some machine learning tasks are intractable with these budgets, limiting the set of tasks available. We are working on tasks with larger budgets, ranging from $1,000 to $100,000. In combination, we are also working on tasks that require multi-node training and significantly more complex infrastructure.

Each agent was only run at one reasoning effort. In future releases, we plan to run evaluations on multiple reasoning efforts to provide a more granular Pareto frontier across total spend.

Agents are currently limited to spending on on-demand AI neoclouds and OpenRouter. We plan to give the agent access to more paid services, such as Tinker and human talent marketplaces.

Link to this sectionConclusion

As agents take on more end-to-end economic work, capital efficiency will become a defining measure of capability. Models must be evaluated on the full cost of completing a task: not just token cost, but also the resources they provision. Resource allocation is a new axis of scaling, where providing an agent with more capital results in better performance.

@misc{frontierml2026,
  title        = {FrontierML: A Challenging, End-to-End Machine Learning Benchmark},
  author       = {{Induction Labs Team}},
  year         = {2026},
  month        = oct,
  howpublished = {\url{https://frontierml.org}},
}