1. TheoPhysEval physics benchmark: what happened?
An artificial intelligence agent has solved 49 of 50 unpublished, research-level theoretical physics problems, according to a preprint posted to the arXiv server on October 9, 2026. The model, GPT-6-Astra, scored 98% when allowed to search the web, a mark the paper's coverage compares to an excellent graduate student. Physicists from ICTS and Google DeepMind led the study.
The paper is titled "TheoPhysEval: Assessing AI agents at autonomous theoretical physics problem solving." It is a benchmark, meaning a fixed set of tasks designed so that different systems can be scored against the same standard. Its particular focus is whether general-purpose AI agents can carry out theoretical physics work from start to finish, without a person guiding them along the way.
The headline number matters because theoretical physics is among the most demanding tests of reasoning. Problems in the field rarely reduce to a lookup or a routine calculation. They require choosing a method, carrying out long derivations and arriving at an answer that can be checked against a known result or a rigorous argument.
2. How were the 50 problems chosen?
The benchmark consists of 50 problems that had never been published. They span 11 subfields of theoretical physics, among them fluid dynamics and quantum field theory. Each is at the level of active research rather than textbook exercise.
Keeping the problems unpublished is the central design choice. Public benchmarks carry a well-known weakness: once questions and answers circulate online, they can end up in the data used to train a model, and a high score may then reflect recall rather than reasoning. Problems that exist nowhere in print are much harder to have absorbed in advance, so a correct answer is more plausibly the product of working the problem out.
The authors also wrote grading criteria meant to approximate what an expert theoretical physicist would require of a solution. Scoring against such criteria, rather than matching a single final number, is intended to reflect how a researcher would judge whether a piece of work is actually right.
3. What did the study ask the AI agents to do?
The goal was to measure what AI agents can accomplish when no human is involved. Nobody prompted the models in the middle of a task, and nobody corrected their mistakes. Each agent received a problem and had to reach an answer by itself.
The researchers tested models that are available to the public, not bespoke research systems built for the occasion. That choice speaks to practical relevance: the results describe what a working physicist could in principle obtain from tools already on the market, rather than what a specialised laboratory prototype might manage.
The work was led by physicists from ICTS and Google DeepMind and involved a large team of authors from various research institutions. Having both an academic institute and an industry laboratory behind the study gives the grading a foundation in physics expertise, while the participation of a major AI developer ensures familiarity with how such agents are built and run.
4. How did GPT-6-Astra compare with the other model?
GPT-6-Astra solved 49 of the 50 problems on the "realistic" track, which allows web search in the way a working researcher would use it. The run on that track cost $116.28 in total, or roughly $2.33 per problem.
The study also ran an "isolated" track in which the agents had no internet access. Under those conditions GPT-6-Astra fell to 46 of 50, so web access was worth three problems. That is a modest gap, and it suggests most of the model's capability came from its own reasoning rather than from retrieving material during the task.
Opus 5.5 scored 46 of 50 on both tracks. For that model, internet access changed nothing at all. One problem went unsolved by every model that was tested, which leaves a small residue of difficulty that neither system could crack.
Taken together, the figures put two systems within a few problems of each other, with the top result separated from the next by three. They also indicate that the isolated condition is a meaningful control rather than a handicap, since performance barely moved without a search engine.
5. Did the AI find errors in the human answer keys?
Yes. According to the study, the models' work led to corrections of some of the human-written reference solutions. That finding is described as the most striking part of the results, because it reverses the usual roles: the answer key, produced by people, was sometimes the element that turned out to be mistaken.
The result carries a practical lesson for anyone building benchmarks at this level. When problems are hard enough, the reference solutions themselves can contain errors, and a model that disagrees with the key may be the one that is right. Verifying such disagreements requires careful expert review, not simple automatic comparison.
It also bears on the memorization question. A model that disagrees with the human reference, and is later shown to be correct, is difficult to explain as a system repeating something it has seen. The pattern is more consistent with genuine derivation, although a single benchmark cannot settle how these systems arrive at their answers.
6. Why does the cost figure matter?
The realistic-track total of $116.28 is small next to the expert time that fifty research problems would ordinarily demand. Even a skilled physicist may need hours or days for a single problem of this kind, and the labor of a team working through a full set would run far higher.
The paper concentrates on performance, not on a complete comparison of costs, so the figure should be read as an indication of scale rather than a finished economic analysis. It does not account for the effort of setting up the tasks, checking the outputs or reviewing disagreements with the reference solutions.
Still, the combination of a near-perfect score and a low price per problem helps explain why research groups are watching results like this closely. If an agent can reliably handle well-posed problems at this cost, the bottleneck in some areas of theoretical work may shift from carrying out derivations to deciding which questions deserve attention.
7. Why do benchmarks like this matter for AI development?
Benchmarks are the main yardstick by which the AI field tracks progress. A model is given a standard set of tasks, its answers are scored, and the results let outsiders compare systems from different developers. As models improve, older tests saturate, with top scores clustered near the ceiling, and new and harder tests are needed to tell systems apart.
Theoretical physics is a natural place to look for such a test. Its problems have answers that experts can verify, yet reaching them takes sustained multi-step reasoning, careful handling of mathematical detail and familiarity with several branches of the subject at once. A system that performs well there is demonstrating more than fluent writing.
Unpublished problems add a further safeguard. Because the questions were written for the study and kept out of public view, the results speak more directly to reasoning ability, and they give other groups a clearer baseline against which later models, including newer releases from the same developers, can be measured in future work.
The distinction between the two evaluation conditions adds to that value. By comparing the realistic track with the isolated one, the authors could separate what an agent knows and reasons out from what it gains by consulting outside material, a difference that single-score leaderboards often hide.
8. What are the limits of the findings?
The study is a preprint, which means it has not yet passed formal peer review. Its claims have been posted for the scientific community to examine, but they have not been through the scrutiny that a journal normally applies. Readers should treat the numbers as preliminary until that process is complete or the results are independently reproduced.
The sample is also limited. Fifty problems is a meaningful set, and the spread across 11 subfields gives it breadth, but it covers only a small slice of theoretical physics. A strong score on these tasks does not guarantee equal performance across the entire discipline.
A deeper caveat concerns the nature of the task. Solving a well-posed problem is different from deciding which problem to pursue. Framing new questions, recognizing which puzzles are worth the effort and judging when an approach is a dead end are central to the work of a theoretical physicist, and the benchmark does not measure them. The agents were handed the questions; they did not originate them.
9. What comes next for AI in physics research?
The immediate next step is scrutiny of the preprint itself. Other researchers can examine the problems, the grading criteria and the disputed reference solutions, and attempt their own tests with the same or newer models. Peer review, if the authors submit the paper to a journal, would add a further layer of checking.
For the benchmark, the open question is how long it stays hard. With one model at 49 of 50 and another at 46, the headroom on this particular set is already small, so future work would probably need fresh unpublished problems to keep discriminating between systems.
For the field, the more consequential question is whether agents that excel at set problems can move toward the harder task of choosing and framing them. The study does not claim that they can. What it does show is that, on unpublished research-level problems with no human help, current publicly available AI agents performed at a level that researchers can no longer dismiss.