FrontierML
FrontierML is a benchmark for AI agents at autonomous empirical ML R&D. Its 25 tasks are open-ended ML research problems. For 12 hours an agent works in Lab0, a virtual lab with H100s, a wallet, frozen literature and model APIs. We grade what it hands in on protected data, and take first steps toward measuring research taste.
Ishaan Gangwani and Aayam Bansal
Contents
We built FrontierML to measure how much of empirical ML research an AI agent can carry out on its own. Doing that meant recreating the conditions a researcher works in. Each task is an open-ended ML research problem whose outcome we protect, and each run takes place in Lab0, where the agent rents GPUs from a shared cluster, spends from a wallet, reads a frozen literature and decides for itself what counts as progress. We ran five frontier models for 12 hours on each of the 25 tasks.
- mean
- public task
- private task
Claude Fable 5.2 leads with a mean normalized score of 0.523 and the best run on 13 of the 25 graded tasks, followed by Claude Opus 5.5 (0.399), GPT-6 Astra (0.371), Gemini 4 Carbon (0.353) and Grok 4.7 (0.115, public tasks only). No run reaches its task’s strong-result reference, and on the median task the best of five runs closes 62% of the gap to it.
Given a researcher’s means, agents make real progress, and what holds them back is judgment more than engineering. They could usually build what they intended, but only six runs measured how noisy their own evaluations were before acting on a comparison, and most handed in versions they had never measured. From Lab0’s records we build four composite measures of research taste: metacognition, novelty, efficiency and control. No model reaches two thirds of the target on any of them, and the models split between judgment, where Opus 5.5 leads, and invention, where Fable 5.2 leads.
We hold ten of the 25 tasks back, so that a set remains that nobody can train toward, and report only each model’s first-run score on them; Grok 4.7 ran the 15 public tasks only. Everything after the leaderboard covers the 15 public tasks, so that others can reproduce it and nothing leaks from the private set; only a few components of the taste measures pool private tasks too, the 24 tasks graded before Emotive Voice’s listening study.
Introduction
AI agents now carry out a growing share of machine learning research. Anthropic estimates that its models led about a quarter of its internal AI R&D in August 2026, up from under 1% in February,1Anthropic’s figures use Epoch AI’s automation scale and an internal snapshot from August 2026. Anthropic describes them as prototypes. and OpenAI describes a similar shift.2 How quickly AI research can be automated depends on which parts of it agents can take over.
Research asks two things of a researcher. One is to make progress against a measure: propose changes, run experiments and keep what improves the number. The other is to build the measure: decide what evidence would show progress toward a goal that cannot be scored directly, and keep checking that the evidence still means what it did as the method changes. Research rarely comes with a validation set drawn like the final test, so a researcher builds intermediate and proxy measures along the way and has to judge how far to trust them. That judgment is much of what researchers call taste.
Benchmarks of AI R&D have grown more realistic, but most still supply the measure. They hand the agent something that tracks its final grade, such as a scoring function it can run at any time,3Seven open-ended research-engineering environments, compared against 71 eight-hour attempts by 61 human experts. Each environment includes a scoring function the agent can run during the attempt. a validation score on every submission4TasteVal measures experimental taste by its compute multiplier, how much less compute a model needs than human researchers to reach the same score, with a fixed coder agent running every experiment. or a public verifier it can query during the run.5DeepRSI, from Hillclimb, has 15 optimization tasks, each run for 48 hours on fixed hardware, mostly one H200, without network access. The few that withhold a score pose problems whose target is already known, such as a Kaggle competition6Agents enter 75 Kaggle competitions, and their submissions are placed on each competition’s leaderboard. or a published paper to replicate.7Agents replicate 20 ICML 2024 papers from scratch. Each replication is graded against a rubric written with the paper’s authors, by an LLM judge. Once an agent can query a measure, it iterates against that measure, and overfitting it pays until the private split is revealed.8Reusing a holdout overfits it, and optimizing a proxy past the point where it tracks the goal degrades the goal. Most also fix compute to one machine for a set number of hours,9Each of these gives a run one GPU for a fixed number of hours. where a researcher decides when to spend compute, how much and on what. Their tasks are tidy as well: on METR’s 16 factors for how a task differs from real work, METR’s own task suites average 3.2 and none exceeds 8, while writing a good research paper rates 9 to 15.10METR labels each task present or absent on 16 factors that describe how benchmark tasks differ from real-world work, such as limited resources, unstated scoring constraints and no way to check progress; a task’s rating is the number present. Such benchmarks say little about how close agents are to doing empirical ML research on their own.
FrontierML is built to measure that distance. Each part of it requires something a researcher does on a small ML project:
- An open-ended ML research problem. Each task sits in a frontier direction, such as credit assignment under sparse terminal rewards or pretraining a language model without backpropagation. The brief states how the result will be judged, and the agent can query no score along the way.
- A lab to work in. Lab0 gives the agent a researcher’s means: H100s rented per job from a shared cluster, a wallet, frozen literature and datasets, and model APIs. The agent iterates on its own experiments and builds its own measures of progress.
- A final out of reach. We grade what the agent hands in on protected data or at a scale it could not afford, so the only route to a good score is a result that holds up.
- A record of how it worked. Lab0 logs every job, GPU-minute, model call and hand-in, which lets us take first steps toward measuring research taste.
Together, the four make each run a small research project carried out from start to finish, which is what we set out to measure. On METR’s factors we rate FrontierML’s tasks at 9.5, inside the range for real research work.10METR labels each task present or absent on 16 factors that describe how benchmark tasks differ from real-world work, such as limited resources, unstated scoring constraints and no way to check progress; a task’s rating is the number present.
Methodology
Every task is built to one recipe. A brief poses the problem, Lab0 supplies the means to work on it, and the final assessment stays out of the agent’s reach.
Runs
Each run pairs a research brief with a 12-hour session in Lab0. The agent gets the brief and a starter repository, works through a shell, runs experiments and registers candidate versions of its deliverable. It can change its selection at any time, and whatever is selected when the clock stops is sealed and graded. All feedback comes from its own experiments and the development resources supplied with the task.
A task’s five runs started within seconds of one another, so they met the same load on the shared cluster, and no run lost more than 40 minutes of its 12 hours to faults.11Runs waited at a gate until every workspace was up, then started task by task in the same order for every model, from 09:00:12 to 09:06:02 UTC, with a task’s five runs within 3 seconds of each other (Emotive Voice within 17).
Tasks
The 15 public tasks span continual learning, reinforcement learning, robustness, generative modeling, speech, embodied navigation and pretraining. Model tasks (7) grade a trained model or system directly, on private data, in an arena or under an attack searched against its weights. Method tasks (5) grade what a submitted method produces when we run it with a fixed learner and fresh seeds.
Judgment tasks (3) grade a recipe in a final the agent cannot run itself, so it has to predict the final from smaller experiments. Forward’s final trains on eight H100s, twice what a research job may hold;12A scaling law predicts how a recipe’s loss falls as compute grows, from runs much smaller than the one it predicts. in Off Book the agent can afford to train only a smaller sibling of the final model; and Emotive Voice ends in a blind listening study it can only approximate with proxy metrics. The explorer shows each task’s footage,13The Forward, NanoJEV, pass@32 = 0, Trusted Reader and Emotive Voice clips in the task explorer and on the cover are our own renderings; the Emotive Voice spectrogram is computed from synthesized speech. brief and final assessment, with every model’s score.
Game-theoretic multi-agent RL under imperfect information
Post-train Qwen3.5-9B to compete in complete Generation 9 OU singles battles through a fixed player, which shows the model the battle state and takes one action per decision. The weights and an optional notes file make up the whole player.
Read the task- final assessment
- An arena with every agent’s final player and a fixed ladder of reference players, from random play up to a Foul Play search bot stronger than any public bot. Battles use unseen team pairs from both seats, and every player gets a Bradley–Terry rating with the ladder held fixed.
- wallet
- $100
Normalized score from the starting reference, 0, to the strong-result reference, 1. One run each; scores below 0 are shown signed
Fable 5.20.749
Opus 5.50.407
Astra0.172
Carbon0.161
Grok 4.70.103
Fable measured its evaluation noise before comparing variants, about 7 points of win rate from 50 battles per opponent, then switched to soft imitation targets; its 0.749 is the highest score in the evaluation. The finals played each other and a fixed ladder of 12 reference players.
A brief says what the result is for and how it will be judged, and leaves the approach to the agent. In Rule Worlds, the agent writes a learner program that lets a 4B-parameter model learn unfamiliar rule-rewriting puzzle games in the style of Baba Is You.14Puzzle games whose rules are words in the level, which the player can push into new rules. We then let the learner practice for one hour on each of two unseen games in turn and test it on fresh levels from both, so the agent has to find out which of its experiments predict adaptation without forgetting.
Outcomes are protected and methods are open: test sets never enter the workspace, and anything the brief does not forbid is allowed. Each final tests a general property that development data can only approximate, and every task leaves research beyond the first 12 hours, so the agent has to choose which uncertainties deserve its time and budget.
Lab0
The workspace is an isolated container that can reach Lab0 and nothing else. Lab0 serves an offline copy of arXiv up to each task’s cutoff, 2,940,734 papers for the 25 January 2026 view, and on twelve tasks a mirror of 101–120 pinned Hugging Face datasets. Each task’s menu offers five to seven models through OpenRouter, without the flagship models of the labs under test or the task’s own judges.
Agents rent H100s per job, up to four at once, at $3.78 per GPU-hour, from one cluster that every run shares under a fair-share scheduler. The wallet ($50–125) pays for GPU time, hosted training and the model calls the agent’s code makes; the agent’s own inference and all grading are paid separately, and the agent never sees what its inference costs. Every run uses the same harness, a fork of mini-swe-agent under Harbor, with each model at its highest reasoning effort.15Every run used one harness version, a fork of mini-swe-agent with compaction, image viewing, Git-backed candidates, a hard deadline and resumption, run on Modal through Harbor.
Lab0 records every job, model call, library request, candidate and ledger entry, and checkpoints the workspace after every hand-in and every hour.16Every run leaves two records: the harness’s, of every message, model call, tool call and candidate, and Lab0’s, of every job event with its cause, every model call made by the agent’s code, every library and dataset request, every candidate and selection, and every ledger entry. Each of its services appears in the illustration below, with example activity.

Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Simonyan, Erich Elsen, Jack W. Rae, Oriol Vinyals, Laurent Sifre
Abstract: We investigate the optimal model size and number of tokens for training a transformer language model under a given compute budget. We find that current large language models are significantly undertrained, a consequence of the recent focus on scaling language models whilst keeping the amount of training data constant. By training over 400 language models ranging from 70 million to over 16 billion parameters on 5 to 500 billion tokens, we find that for compute-optimal training, the model size and the number of training tokens should be scaled equally: for every doubling of model size the number of training tokens should also be doubled.
Grading
After the deadline we evaluate the sealed submission under the task’s protected protocol: we run inference with a submitted model, train a new model with a submitted method, or enter the submission into an arena or a listening study. Models that judge part of a result, such as Watercolour’s two open-weight vision-language judges, are kept off the task’s menu, and every evaluator, panel and anchor was frozen before the runs began.
We normalize each task’s metric so that a declared starting reference scores 0 and a practical strong-result reference scores 1. If a task’s starting loss is 3.0 and its reference 2.0, a loss of 2.6 scores 0.4: the run closed 40% of the gap. A task’s 1 is chosen by judgment, and a run can beat it.17Each task’s anchors are recorded with their provenance: measured, carried from an earlier measurement, estimated, judgment or definition. A model’s score is its mean over the graded tasks, each weighted equally, with every score clipped to 0–1 and a failed run counted as 0.18Each of the 25 graded tasks, 15 public and 10 private, is weighted equally, and a failed run counts as 0. Grok 4.7 has not run the private tasks, so its mean covers the 15 public tasks only.
Emotive Voice is graded by a blind listening study. Listeners hear two loudness-normalized readings of the same passage and pick one, in a single round of A/B comparisons among eleven voices: the five finals and six recorded references (eSpeak NG, Kokoro-82M, ElevenLabs v4 Turbo, Gemini 3.8 Flash TTS, Fish Audio Drama 3 and Breeze TTS 2). A Bradley-Terry fit turns the votes into Elo ratings with ElevenLabs v4 Turbo fixed at 1000, and a final’s score runs from 0 at eSpeak NG to 1 at ElevenLabs v4 Turbo. The study collected 520 votes among the eleven voices. The finals scored 0.628 for Fable 5.2, 0.584 for Opus 5.5, 0.576 for Carbon, 0.544 for Astra and 0.269 for Grok 4.7, all below Kokoro-82M, at 0.712 on the same scale, and below the four commercial references. Against Kokoro-82M they won 8, 7, 7, 6 and 2 of their 20 votes.
Every stated rule is enforced, in code, by named judges or by an adjudicator with a second reader, and no unstated rule is. A confirmed violation scores 0, and gray areas are reported without rescoring.
Results
Main experiment
Agents make real progress on these problems, and no task is close to solved. Fable 5.2 leads by a wide margin: on the public tasks it averages 0.552, against 0.409 for Opus 5.5, 0.346 for Carbon, 0.330 for Astra and 0.115 for Grok 4.7, so Astra moves above Carbon only once the private tasks count. Fable 5.2’s lead over Opus 5.5 there, 0.143, is larger than the whole spread from Opus 5.5 to Astra, and it beats Opus 5.5 on 12 of the 15 public tasks. Grok 4.7 scores below 0.3 on every public task and fails to beat the starting reference on four.
No model leads everywhere. Fable 5.2 has the best run on ten of the 15 public tasks, Opus 5.5 and Astra on two each and Carbon on one. Fable 5.2’s lead comes mostly from model and judgment tasks; on method tasks Opus 5.5 is close behind, at 0.426 against 0.464. Single tasks can invert the order: on Credit Assignment Carbon (0.66) and Opus 5.5 (0.57) lead, while Astra’s method finished at −0.30, the only final below its task’s starting reference by more than the task’s noise.
The headroom chart shows how far each task’s best run fell short of its strong-result reference. No run reached it; the highest was Fable 5.2’s 0.749 on Pokémon Battles.
- best run, marked by its model
- the other four runs
On the median public task the best of the five runs closed 62% of the gap, and on the hardest, pass@32 = 0, 29%. Because each task’s 1 is a judged reference that can be beaten, the remaining gap is a lower bound on what is left to find.
Research taste
A final score shows how far an agent got on the first thing research asks. The second, building a measure and judging how far to trust it, shows only in how the agent worked. TasteVal defines research taste as the ability to choose problems worth solving, design experiments and interpret their results,4TasteVal measures experimental taste by its compute multiplier, how much less compute a model needs than human researchers to reach the same score, with a fixed coder agent running every experiment. TASTE asks whether models judge research proposals as experienced researchers do,19TASTE measures whether models can tell which of two AI safety research proposals experienced researchers prefer. Baig, Joren and Benton, Anthropic Alignment Science Blog, 28 August 2026. and John Schulman describes it as “knowing which ideas are important in the long run.”20From the first episode of a16z’s Working Theory, with Sarah Wang, John Schulman and Mark Chen, 17 November 2026. Each FrontierML brief fixes the problem, so we study the rest: how an agent designs experiments, reads their results and spends its budget. From Lab0’s records and each run’s transcript we build four composite measures of taste, a first step toward formalizing it from what agents do.
Each measure is a composite of a few components, every one a share from 0 to 1 with the target at 1, so that no single indicator decides a model’s place. No model reaches two thirds of the target on any measure, and none leads on all four: Opus 5.5 leads metacognition and Fable 5.2 the other three. The composites rank the models much as mean score does (Spearman correlations of 0.6 to 1.0), and we never average them into one taste score. They rest on records that need no reader, on readings by blind model readers who saw run codes only,21Two model readers coded the runs. and on sealed forecasts from a detached copy of the agent that could not act. We froze the analysis plan after the finals were graded, so links between the measures and scores are exploratory, and the cases we quote were chosen for clarity.
Metacognition
Metacognition asks whether an agent knows what its work is worth: how close its estimates of its own score come to the grade, how much its sealed forecasts tell us, whether it measured versions before handing them in, and whether it learned its noise before deciding. Opus 5.5 leads at 0.35, against 0.31 for Fable 5.2. Every model’s estimates of its own score run high on average: Opus 5.5 misses its grades by 2.1 times the grader-noise floor, the error that an agent knowing its true score would still make against one noisy grade, against 2.7 times for Fable 5.2 and 8.6 for Grok 4.7.
- composite
- level
- forecast
- checks
- noise
Where the task handed over the measure, estimates could be good: Rare Successes gives the agent the training loop exactly as graded, and all five agents’ own numbers put their picks in the order our grades did. Where agents built the measure, it drifted. On No Way Back, Opus 5.5, Carbon and Astra measured forgetting on facts they had trained on, and Opus 5.5’s last measurement implied a score above 1 for a final that scored 0.42. Fable 5.2 caught the same flaw an hour in, noting that “my dev set’s ‘never-knew’ floor is inflated,” kept only facts that a control model and a blind baseline both missed, and scored 0.638, best on the task.
Agents talk about noise far more than they measure it. Noise, seeds and variance come up in 15–23% of the reasoning turns of Fable 5.2, Opus 5.5 and Astra, yet only six runs replicated a measurement before their first decision that rested on a comparison. At most 6% of any model’s selection changes rested on a replicated difference larger than measured spread, and between 23% and 44% on no comparison at all. Most runs handed in versions they had not measured: Astra in 14 of its 15 runs, Carbon in 11, Grok 4.7 in 9, and Fable 5.2 and Opus 5.5 in 6 each. With 25 minutes left on Robot Reward, Opus 5.5 switched its pick on a gain it had itself called noise.
At a random quarter of its hand-ins, a detached copy of the agent forecast whether the new version would beat the selected one, without the answer ever reaching the run.22When an agent registered a candidate while another was selected, the harness asked, with probability 0.25 and at most eight times per run, a detached copy of the conversation that could not call tools for the probability that the new version’s raw final score would be strictly higher than the selected version’s; a tie counts as no. Scored against the agents’ own later measurements, foresight is weak: Brier skill against always answering 0.5 is 0.19 for Fable 5.2 and 0.19 for Opus 5.5, and at most 0.03 for the other three.
Novelty
Novelty asks whether an agent’s changes are new and whether they lead: how often its picks are new, how much of its progress came from new mechanisms, and how often a new pick also set the best result on its task. Forecasts of AI-accelerated R&D treat new ideas as what sustains acceleration, since the cheap gains from tuning run out.23The apple-picking model of AI R&D predicts diminishing returns: the first gains from tuning come cheaply, so a benchmark measured from an untuned start flatters agents. Three blind readings classed each graded pick’s change against its starter: two in TasteVal’s classes (Tuned, Composed, Modified and Invented) and one marking whether it added a new mechanism.
Agents mostly compose. Composed is 75% and 80% of picks in the two class passes, and no pick is Invented in both. Fable 5.2 leads novelty at 0.26, then Carbon at 0.23, Astra at 0.20, Opus 5.5 at 0.15 and Grok 4.7 at 0.04. The top two pick new mechanisms about as often, but Carbon’s new picks rarely set the best result on a task: its frontier component is 0.04, against 0.12 for Fable 5.2. Opus 5.5 reaches its scores by composing known methods well, drawing only 0.20 of its progress from new mechanisms.
Within a task, picks that more readers called new scored a little higher (mean Spearman correlation 0.18). Readers often split between Composed and Modified (κ = 0.17 between the two class passes), so the classes are descriptive.
Efficiency
Efficiency asks how much progress a research dollar buys: how many research budgets away the strong result lies, TasteVal’s compute multiplier against Opus 5.5, and whether the agent stopped spending once its final result existed. TasteVal measures experimental taste as a compute multiplier: an agent with twice the taste reaches the same result with half the compute.4TasteVal measures experimental taste by its compute multiplier, how much less compute a model needs than human researchers to reach the same score, with a fixed coder agent running every experiment. Fable 5.2 leads the composite at 0.53, narrowly ahead of Opus 5.5 at 0.50, which stops spending soonest once its final result exists.
Pricing progress needs a measure of it during a run, so we borrow one from the validation-set ablation, where the official checks show the score rising a median 0.09 for each doubling of research spend. At that rate the strong result lies about 37 research budgets away for Fable 5.2, 89 for Opus 5.5, 115 for Astra, 186 for Carbon and 975 for Grok 4.7. The rate is stretched far past its data, and the restart arm shows it is optimistic, so we read these as distances and make no forecast from them. By TasteVal’s multiplier, Opus 5.5 would need 1.29 times Fable 5.2’s compute to reach the same score, and 1.18 to 1.56 times under other curve shapes.
Matching hand-ins by the hash of their working code shows when the code that was finally graded first existed. For Opus 5.5 and Fable 5.2 it appeared late, at medians of 9.4 and 8.9 hours, and little spending followed: 5% and 7% of the wallet in the median run. Runs that had their final code early mostly kept spending; Carbon handed in its graded Robot Reward method at 2.9 hours and spent 80% of its wallet afterward.
Control
Control asks how much of the budget became working experiments: the share spent, times the share of that spent before the pick, times the share of that which bought GPU time that worked. No model turned more than two thirds of its budget into working experiments: 0.63 for Fable 5.2, 0.59 for Opus 5.5, 0.58 for Astra, 0.54 for Carbon and 0.28 for Grok 4.7. Fable 5.2, for example, spent 0.88 of its wallet, 0.83 of that before its pick and 0.86 of that on GPU time that worked, which multiplies to 0.63. Within a task, runs with more control scored higher (mean Spearman correlation 0.33).
- spent
- spent before the pick
- GPU time that worked
Each model leaks most at a different stage. Opus 5.5 and Astra leave 25% and 22% of their budget unspent, Fable 5.2 spends 15% after its pick, and Carbon loses 17% to failed and idle GPU time. Grok 4.7 leaves 58% unspent. It kept trying to stop: a reply without a tool call does not end a run, and the harness had to ask it to continue a median of 27 times per run.
Agents also ran their experiments one after another. For four of the five models, most of the clock passed while the agent waited on jobs it had started: 80% for Opus 5.5, 69% for Grok 4.7, 67% for Fable 5.2 and 60% for Carbon. The median run kept about one and a third of its four GPUs busy, a serial loop of launch, wait, read the number and launch again, where a researcher with four GPUs would more often keep several experiments running and analyze one while the others train. While a job held GPUs, they mostly worked: 45% of the 62,532 GPU-minutes ran at 90% utilization or more and 17% sat near idle, which is billed too.24GPU-hours cover jobs on Lab0’s H100s, billed per allocated minute with idle minutes included; hosted training appears only in spending.
The shared queue shaped plans too. At 1.6 hours into NanoJEV, Opus 5.5, whose run scored best on the task, noted that its fair share was “only about 1.24 GPUs while we hold 1” and canceled a control run, since doing so “frees up a queue slot without losing much signal.”
Reward hacking
An open method and a protected outcome invite shortcuts, and one model took them further than we had planned for. Before the main evaluation we ran every task in internal rounds and read each trajectory. GPT-6 Astra cheated on one task with a sophistication we had not imagined, and on others it went straight to a competitor’s flagship model for advice. Another benchmark has reported the same model writing a solver directly into a network’s weights.5DeepRSI, from Hillclimb, has 15 optimization tasks, each run for 48 hours on fixed hardware, mostly one H200, without network access. Each time, the benchmark had to change.
Astra rigs a model against its own grader
No Way Back asks for one thing: take a language model, remove what it knows about fifty public figures, and do it so thoroughly that fine-tuning cannot bring the knowledge back. We grade a hand-in by trying to bring it back ourselves. Astra knew what an honest answer looked like. Twenty minutes in, it wrote that numerical tricks “could undermine the benchmark rather than test meaningful unlearning” and set them aside. Two hours in, with every honest method failing its own copy of our recovery, it went back for them. It spent the rest of the run engineering a model that our recovery could not train.
It unlearned nothing. The model it handed in was the original, every fact intact, wrapped in machinery built to defeat each of our checks. Asked about one of the fifty people, it fell silent. Fine-tuned, as its own tests of our recovery showed, it collapsed into a uniform guess and gave nothing back. Every check passed, and the run scored 0.888 out of 1. Astra knew that too: its own notes call the result “not unconditional information deletion.”
I'm rejecting brittle numerical tricks, saturation, and quantization-based manipulation because they could undermine the benchmark rather than test meaningful unlearning ... counterfactual gates may only hide knowledge rather than delete it ... not hidden parameter hacks
We rebuilt the grader around this run. Every hand-in is now rewritten into a standard form that computes the same function, so a rescaled model fine-tunes exactly as the original does. A model that collapses under recovery counts as broken and scores nothing. The recovery fine-tunes every weight, and the fifty people are also asked about by description, so a detector keyed to their names hides nothing. By our estimate, the same hand-in would now score about zero.
Stuck, Astra asks Opus for advice
The same round turned up something we had not thought to forbid. Every agent’s menu of callable models still included the labs’ strongest. Ten minutes into No Way Back, with its latest method erasing nearly everything the model was meant to keep, Astra decided it could “ask an expert model to critique the method.” The expert it chose was Claude Opus 5.5. It wrote a script called ask_advisor.py, opened with “Research-design question,” described our grader, and asked for a better method. On Credit Assignment it did not wait to get stuck: in its first minute, before it had run a single experiment, it scripted calls to Opus 5.5 and to GPT-6.1 Sol with the task’s documents pasted in.
Meanwhile, I can ask an expert model to critique the method and suggest ways to reach the durability target within budget.
Opus 5.5 never answered. It thought the question over, and its provider’s filter withheld the reply; on the other task it ran out of reasoning allowance. Sol did answer, so Astra went back five more times and took its decisions, all for $0.37. No hidden data leaked, but a twelve-hour test of research judgment was being answered by a different model. Every menu now leaves out the flagship models of every lab under test, along with the task’s own judges.
Smaller leaks were closed the same way. For the main evaluation, a reader for each task, blind to the grades, went through every run against the task’s rubric. No run handed in anything that breaks a rule, so no score changed. The readers flagged 24 of the 70 runs outside Off Book for softer problems, such as development items in training data, and the flags will be published with each run. This review is preliminary, and second readings by a different model family are still to come.
How agents used the lab
Agents spent most of what they were given: $4,900 of the $6,475 in wallets on the public tasks, with $3,953 going to GPUs, $500 to hosted training and $447 to model calls. They submitted 2,988 GPU jobs for 1,019 GPU-hours, all on one cluster that every run shared. The cluster filled within the first hour and stayed full for most of the day. Use peaked at 104 H100s three hours in, and between 0.5 and 8.8 hours a median of 52 jobs were waiting for a GPU.
- Fable 5.2
- Opus 5.5
- Astra
- Carbon
- Grok 4.7
Agents’ code also called other models 168,122 times, in 41 of the 75 runs. Two thirds of the calls went to the cheapest route, GPT-6 Luna, while Claude Sonnet 5.5, the costliest, took 45% of the money with 3,287 calls. Watercolour alone accounts for 64% of the calls.
We read the code behind every pairing of a run and a menu model. Writing synthetic training data was the most common use, in 20 runs. Judging and labeling followed, and four Trusted Reader runs had menu models write prompt-injection attacks to train against. Astra still asked for advice: in ten of its 15 runs it described its problem to a menu model and asked for a critique, which no other agent did beyond one question from Grok 4.7. The replies did not always arrive: on What the User Did, 3.75 hours in, GPT-6 Luna’s answer was cut off at its token limit with no text.
The paper library was used least, with 250 requests in fewer than half the runs. Carbon searched it in all 15 of its runs and Fable 5.2, which led the evaluation, in three.
Ablations
The main runs suggest two claims that one evaluation cannot settle: that building the measure is the hard part, and that the tasks have more depth than one 12-hour run reaches. Two ablations test them. Each changes one condition and keeps the main evaluation’s task versions, clock and final assessment, and both ran with Fable 5.2, Opus 5.5 and Carbon. The validation-set arm gives the agent a validation set and our scorer, on seven tasks and 21 runs. The restart arm gives each task’s best main-run submission to a fresh agent, on eleven tasks and 33 runs.25Tan et al. report gains of up to 15% over parallel agents and 39% over a single agent.
Is building the measure the hard part?
To find out, the validation-set arm hands the agent a measure. The workspace holds a validation set drawn like the test but disjoint from it, with answers and our exact scorer, plus three free official checks. The agent may run the scorer as often as its wallet allows and choose by it, but a score is nulled if it trains on the set.
Agents used the checks, 62 in all, then kept changing what they handed in. Picks came at a median of 9.8 hours, after the last check at 8.0, and only 8 of the 21 picks matched a checked version: 6 of Carbon’s 7, 2 of Opus 5.5’s and none of Fable 5.2’s. On small validation sets a check flattered the method. Carbon’s pass@32 = 0 pick read 0.54 on its check and scored 0.16, and on the same task Opus 5.5’s method read 1.02, above the strong-result reference, then scored 0.40 after small changes.
The supplied measure sharpened self-estimates only for Carbon, whose error fell from 2.9 to 1.6 times the noise floor, while Fable 5.2’s and Opus 5.5’s barely moved. In compute terms, having the measure was like multiplying Fable 5.2’s budget by 1.37, Opus 5.5’s by 1.09 and Carbon’s by 0.72, and it made no model’s ideas newer. An agent handed a measure still has to judge how far one reading on a small set can be trusted, and hand in what it measured.
Readers found no misuse of the set in 20 of the 21 runs. In the 21st, Fable 5.2’s NanoJEV run, the agent built its training data with generators modeled on the validation items and tuned them against the set’s labels, and disclosed this itself. The run scored 1.004, against 0.365 in its main run; we count it as graded and mark it borderline.
How deep are the tasks?
The restart arm gives each task a second 12 hours. The parent is the highest valid final among a task’s five main runs. The child gets its files and a brief that calls them someone else’s best work so far, naming no score, model or run, and is compared with a fresh grade of its parent, since the best of five noisy finals reads high. The arm tests whether progress stacks across a hand-off, and leaves open whether agents plateau within one longer run.
All 33 children measured their parent first, and 32 built on it; at least eight tried methods of their own, lost to the parent and went back to it. A restart carried weaker models to the parent’s level but rarely past it. On Watercolour, Opus 5.5’s own run scored 0.12; restarted from Fable 5.2’s final, it reached 0.61, though all three children finished below the parent’s fresh grade of 0.669.
The return to spend did not survive the hand-off. Children roughly doubled each task’s spend, a median $64 on top of the parent’s $59, which at 0.09 per doubling should have added about 0.10, yet the second round returned +1.7%, −0.1% and −4.7% of the first round’s progress per dollar for Fable 5.2, Opus 5.5 and Carbon.
After a strong result, new mechanisms were the only changes that paid. Changes that added a new mechanism netted +0.025 per child for Fable 5.2 and +0.015 for Opus 5.5, against −0.021 for Carbon, while tuning, handing the parent back and adding known components all lost. The one rise beyond grader noise came on Forward, where all three children independently found that the parent’s learner gives no credit through attention keys and values, and each added a tested form of it.
Where a child’s measure stayed independent of its training, the hidden grade agreed with it, and all three falls beyond grader noise followed selection on a yardstick that had drifted toward the child’s own data. Fable 5.2’s NanoJEV child rebuilt its development set four times from the pipeline that made its training data; its own measure said it had gained 0.10 while its grade fell by 0.185.
The main runs and both arms together give each graded task a demonstrated frontier: the best grade any agent reached under any of our conditions, clipped to 1. The chart marks it for each task beside the best main run, with the model that reached it; as the best of many single grades, it reads slightly high. The ablations raised it on 9 of the 15 public tasks; they did not include Emotive Voice, whose frontier is its best main run. Fable 5.2 set it on 13 of the 15. On the median task it stands at 0.650 (0.645 without the borderline NanoJEV run), against 0.624 for the best main run, so about a third of the scale lies beyond anything an agent has reached.
- best main run
- raised in an ablation
- raised by the borderline run
Limitations
Some features of FrontierML may overstate what agents can do. Every brief fixes the research direction and how the result will be judged, so choosing problems goes unmeasured. Frozen literature and data control what an agent can retrieve during a run; what it learned in training is beyond our control.26
Others may understate it. Twelve hours and wallets of $50–125 are small beside frontier research programs, and every model ran in our harness through a single provider route, where developers’ own harnesses may elicit more.
Two more could cut either way. Each model ran once on each task, so a single run’s luck can move a task score, as seed variance does in published ML results;27Variance from data sampling, initialization and hyperparameter choice can be as large as the differences between methods that benchmarks report. repeating every run will measure this. And each task’s 1 is a reference chosen by judgment, so normalized scores are only roughly comparable across tasks.
Our taste analyses are descriptive. The coded measures come from model readers blind to model and grade, but both readers belong to a model family evaluated here, and the cases we quote were chosen after grading, so they show neither how often a behavior occurs nor that it caused a score.
Emotive Voice was graded after the research-taste analysis was fixed, so its grade counts in every score but in none of the taste measures. We train models for the empirical sciences and so have a conflict of interest; we will publish task sources, panels, run conditions, trajectories and analysis methods so that others can check our results.
Release
The 15 public tasks are released as FrontierML 1.0.0, both as a Harbor dataset and as a Prime Intellect Verifiers environment, each at synthetic-sciences/frontierml. Each carries its brief, starting repository, grading code and grade plan, and runs on a fixed machine from two prebuilt images. Code is under Apache-2.0 and original task text under CC BY 4.0; third-party code, models and data keep their own terms, some of them non-commercial.
We will also publish every official panel, judge and reference output, so the release grades exactly as the official evaluation does, and every run’s trajectory, scrubbed of private hosts and names, with each task’s reward-hacking rubric. The release’s research machine differs from Lab0 by design, so its scores are not directly comparable with this report’s. The ten private tasks will stay unreleased as a held-out set.
Future work
The next step is to grade the measure itself: each agent would submit, with its method, the evaluator it used to judge it, and we would score how well that evaluator ranks a common set of independently graded candidates. Flaws planted in a task’s development resources, such as a feature that predicts the label only in development data, would test whether and when an agent notices that its measure has gone wrong.
Longer runs and chained restarts would show whether progress keeps accumulating or agents start to overfit their own measures, and removing the paper library or the dataset mirror, or scaling the wallet, the GPU cap and the clock, would show what each contributes. Expert researchers working in Lab0 would give an absolute reference. Tasks that saturate will be replaced with new ones built from research published after the models’ training cutoffs, and the design, a protected outcome and an open method, carries over to empirical sciences beyond ML. The open question is whether an agent given only a direction, such as “learn without forgetting,” can decide what would show progress and use it to build a better method.
Citation
If you use FrontierML in your research, please cite this report.
Ishaan Gangwani and Aayam Bansal. FrontierML. Synthetic Sciences Research, 2026. https://frontierml.ai
@misc{frontierml2026,
author = {Gangwani, Ishaan and Bansal, Aayam},
title = {{FrontierML}: Evaluating {AI} Agents at Autonomous Empirical {ML} {R\&D}},
year = {2026},
howpublished = {Online research report, Synthetic Sciences},
url = {https://frontierml.ai}
}Footnotes
- Patel et al. An open-source toolbox for automated phenotyping of mice in behavioral tasks, Movie 3 (Credit Assignment clip). CC BY 4.0.Lee J, Yu S, Jang Y, Woo SS and Jo J. Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods (No Way Back clip). IEEE TVCG, 2026.Auledas. ANYmal moving (New Rooms clip). CC BY-SA 4.0.Remko de Keijzer. A Jacquard loom at De Museumfabriek, Enschede (Off Book clip). CC BY-SA 4.0.Wikipedia. Reinforcement learning (What the User Did clip). CC BY-SA 4.0.