SemiAnalysis GDPx: Why the Best AI Model Scores Just 12% on Real Work

SemiAnalysis just launched GDPx, a private benchmark of 206 expert-built agentic tasks across 11 high-value domains, and the best frontier model, Claude Opus 5.5, scores only 12%. In episode 42 of the SemiAnalysis podcast, Jordan Nanos sits down with GDPx builders Max Kan and Andrew Megalaa to explain why most popular benchmarks are quietly broken, why they picked chip design, cyber and biology over insurance and real estate, how a real private equity deal room became a finance test, and what the leaderboard says about Opus 5.5, GPT-6 Astra, Gemini 4 Carbon and Meta’s Muse Wonder Full. The benchmark went live on Sunday, November 1, 2026, with this launch thread:

Today we’re releasing GDPx, a benchmark for measuring agentic capabilities across the most valuable domains.

GDPx is a true frontier-level benchmark: even the best model scores just 12% (1/7)🧵 pic.twitter.com/RNnkcuUkih
— SemiAnalysis (@SemiAnalysis_) November 2, 2026

TLDW

GDPx is SemiAnalysis’s answer to benchmarks that are hard because they are unfair, not because they are difficult. Fourteen specialist data partners built tasks in chip design, finance, materials science, hardware design, legal, cybersecurity, alignment, software engineering, biology, recursive self-improvement and quant trading, and SemiAnalysis QA’d every one and dropped more vendors than it kept. Opus 5.5 leads overall at 12.0%, but safety classifiers distort the ranking: GPT-6 Astra scores 0% on cyber because it refuses, while Muse Wonder Full tops cyber at 35.2%. The team also defends binary grading, explains why only one task per partner is public, and argues about what any of this means for jobs.

Thoughts

The finance task is the best argument in the whole episode for why GDPx is different, and it quietly demolishes the “AI has already automated finance” story. Instead of asking a banker to invent a realistic task for two hours, the partner took a real private equity deal (a firm buying about 62% of a family-owned snack company for a little over $800 million), with all 4,000 files split across the PE firm’s, the bank’s, the accountants’ and the seller’s own data rooms, organized and disorganized exactly as they really were. They even replay the data room as it looked on the day each task was assigned. Under binary grading every model scores zero except Gemini 4 Carbon, which got one task right. What fails them is not reasoning. It is diligence: one email telling the associate to adjust the numbers to Q4, and a model that did not look hard enough ships the wrong figures. That is the gap between a demo and a job, and it is the kind of failure a human associate would be yelled at for. Anyone selling an “AI analyst” should be asked how their product does on a task like this one.

The public versus private trade-off is where the benchmark business gets honest about itself. Max Kan says it bluntly: make a benchmark public and it stops measuring capability and starts measuring which lab bought the most training data shaped like your tasks. That is Goodhart’s law with a procurement budget, and it is a useful lens on every leaderboard you read. GDPx’s answer is to release one task and its rollouts per partner and keep the rest private. The catch is that a private benchmark asks you to trust the builder, and here the builders are SemiAnalysis plus fourteen data companies whose business is selling RL environments to the same labs being scored. The team leans on its own agentic QA (the same kind that flags roughly half of Terminal Bench 4 as broken) to earn that trust. It is a reasonable compromise, but readers should hold the 12% headline with the same skepticism the hosts apply to everyone else’s numbers.

The safety classifier finding matters more for buyers than the headline ranking does. GPT-6 Astra scores 0% on cybersecurity, not because it cannot do the work but because it refuses it, and GPT 6.1 Sol does the same. Meanwhile Meta’s Muse Wonder Full scores 35.2% on cyber, well ahead of the next model. Strip out cyber, biology and alignment and the order flips: Astra 11.8%, Opus 11.7%, GPT 6.1 Sol 10.5%. SemiAnalysis chose the all-domains chart as the headline because it is “fair to penalize overly strict safety classifiers to some degree,” and that is the right call. What you can actually deploy is the model plus its refusal policy, and the hosts describe Anthropic’s lower-tier cyber access program as more or less useless because you still get flagged on basically everything. For security teams, the practical takeaway is simple: the best cyber model today is whichever one will do the job.

The thesis behind the domain list is the most investable idea here. SemiAnalysis argues the first trillion dollars of AI revenue will concentrate in a few sectors that keep paying for smarter models, not diffuse across the economy, so a benchmark weighted by GDP share or occupational taxonomy measures the wrong thing. Finance and legal earn their places through margins and competition: these businesses can afford huge token bills, and in a lawsuit or a contract redline, having a better AI than the other side is a direct win. That is a sharper answer to the “most tasks don’t need frontier models, so the AI trade is over” argument than most of the bull case manages. Both claims can be true at once. Day-to-day tasks get cheap and commoditized, while chip design, biology, cyber and adversarial professional work keep buying Fable 6, 7 and 8.

The jobs conversation is the most candid part of the episode because the hosts openly disagree. Max describes a “Goldilocks zone” where Opus is a better software engineer than most engineers on a well-specified problem, yet good engineers are worth more than ever because they supply the taste and judgment. Andrew pushes back that most people just want to put their hours in and go home, and that AI may force them to care. The intern detail is the tell: a quant firm’s interns finished their projects in the first few weeks, and the overhead of managing an intern now often loses to telling Claude to do it. The materials science partner Bryan made the same point from the other direction, watching Astra beat a published state-of-the-art reference on a subtask. The lowest rung of expert work is the first to go, and the job left over is the one that asks for ambition.

Key Takeaways

GDPx has 206 tasks across 11 domains, built with 14 specialist data partners, with chip design (34 tasks) the largest domain. Each domain and each partner within a domain is weighted equally.
Claude Opus 5.5 leads the overall leaderboard at 12.0%, followed by Gemini 4 Carbon (10.9%), GPT-6 Astra (9.6%) and Muse Wonder Full (9.0%).
Cost per task varies wildly: Opus 5.5 runs about $51 per task, Gemini 4 Carbon about $16, GPT 6.1 Sol under $10, and DeepSeek 4.1 Flash under $1 for a 5.4% score.
Excluding cyber, biology and alignment, where safety classifiers kick in, GPT-6 Astra edges ahead at 11.8%, with Opus 5.5 at 11.7% and GPT 6.1 Sol at 10.5%.
Muse Wonder Full is the best cyber model available by a wide margin at 35.2%, while GPT-6 Astra and GPT 6.1 Sol score 0% because they refuse the tasks.
Many popular benchmarks are “difficult” only because they are unfair: a verifier that only accepts one of two valid fixes marks a correct model as wrong. Data companies have found that a large share of Terminal Bench 4 tasks are broken.
SemiAnalysis dropped more data providers than it included because many failed QA, and tasks were still being finalized the Thursday before the Sunday launch.
The domains were chosen for verifiability and for their capacity to absorb ever-smarter frontier intelligence at high ROI, which is why chip design, cyber and bio made the cut over government, real estate and insurance.
The finance environment is a real private equity deal with about 4,000 files across four data rooms, and the real tasks analysts and associates were assigned, replayed at the moment each was given.
Finance and legal use binary grading because a half-right workbook is a failed deliverable. Finance is nearly all zeros under binary grading, and the best partial scores sit around 50% for finance and in the 30s or 40s for legal.
Most other domains use continuous grading, for example scoring a chip design PPA optimization between a naive solution and a theoretical maximum.
Human baselines no longer make sense for some tasks: Astra beat a published state-of-the-art reference on a materials science subtask, and some chip design tasks would take whole teams.
Gemini 4 Carbon marks a turnaround for Google, putting it at the top of the “tier two” cohort with Meta and xAI, though the hosts still see a meaningful gap to OpenAI and Anthropic.
Only one task and its rollouts per partner are public, to stop labs hill climbing the benchmark by buying lookalike training data.

Chapters

0:10 Launch Week Chaos

The team had no idea what the final leaderboard would look like until a few hours before publishing. Partners had difficulty-calibrated tasks on only a handful of models, and full rollouts plus grading could take up to 24 hours each. Andrew Megalaa went on polyphasic sleep to build the infrastructure, with Claude set up to wake him if something broke. They rushed to launch on a Sunday because new OpenAI and Anthropic flagships were expected that week, and you cannot launch a benchmark the same day as Fable 5.5.

4:02 Broken Benchmarks: Difficult vs. Unfair

Max Kan argues that many popular benchmarks are made by people without expertise in evals or RL environments. The classic failure is a coding task with two valid fixes where the verifier only accepts one, so a model that rightly picks the other is marked as failing. The hosts name Terminal Bench 4, SWE-bench Pro and DeepSWE as examples with these issues. Everyone in the labs already knew, they say; nobody spent the time to write it up.

7:33 Building GDPx and Choosing the Domains

SemiAnalysis wanted its own eval so it could say something credible about each new model release without testing every one by hand. Rather than write 200 tasks itself, it partnered with specialist data companies and kept final QA. Domains were picked for verifiability and for their ability to absorb frontier intelligence, refined over several weeks of conversations with lab researchers. Some partners are established data companies, and one materials science partner is essentially a single very strong PhD writing tasks from papers they and their colleagues wrote.

15:00 Models vs. Experts

Bryan, the materials science partner, was surprised that GPT-6 Astra solved tasks he expected it to fail, and had to build a harder set. Andrew Megalaa describes his own version of that moment a year earlier, as Sonnet and Opus releases convinced him he would stop writing code by hand. He still believes a human plus AI beats AI alone, because a person supplies entropy the model lacks. Human baselines stop making sense when a model beats a published state-of-the-art reference, and some material science tasks cost the models about $10 to $50 each.

20:44 The Leaderboard, Cyber and Safety Classifiers

The team debated whether the headline should include cyber, biology and alignment, where safety classifiers sandbag some models. They kept all domains, arguing it is fair to penalize overly strict classifiers. Jordan Nanos had independently found Muse Wonder to be a top cyber model in his own work, which the results confirmed. The hosts call the labs’ cyber access programs confusing, with lower tiers that still flag nearly everything.

27:13 The Finance Task: A Real Private Equity Deal Room

Max contrasts GDPval-style task creation, where an expert invents a task in a couple of hours, with GDPx’s finance partner, which rebuilt a real snack company acquisition from its actual files and emails. The models were given the real assignments from the deal, timed to when each was handed out. Failures were often about missing a single email with a key instruction. The team’s finance friends confirmed an analyst would have been yelled at for the same miss.

31:27 Binary vs. Partial Grading

Finance deliverables like a quality of earnings workbook break down into atomic criteria, but GDPx still grades them pass or fail, because a half-right workbook is useless to a boss. Max acknowledges critics who say binary grading hides easy tasks, and offers a test: a benchmark at 25% binary and 97% partial is probably broken, while GDPx finance sits around 50% partial and legal in the 30s or 40s. Most other domains use continuous grading, such as chip design optimization scored against a theoretical maximum. He also argues finance and legal will absorb heavy token spend because of their margins and adversarial dynamics.

36:48 AI and Jobs

GDPx deliberately skips the white-collar work people fear losing most, like customer service, so the hosts treat the jobs question as mostly separate. They describe a Goldilocks period where human taste plus models beats models alone, while warning that one supervisor per company is a possible future. Interns are the first casualties, since their well-specified work now goes to Claude. Jordan notes rising CVE counts as one sign of growing demand for cybersecurity skill, and the group splits on whether people will rise to the moment.

43:44 Google’s Comeback

Andrew says Opus 5.5’s showing matched his subjective view, while Astra was clearly a step ahead in materials science and some engineering work. Gemini 4 Carbon, which Sundar Pichai quote tweeted, reverses a trend where Google seemed to be falling behind Meta, xAI and even Chinese open models. Still, on the apples to apples view without the safety-heavy domains, Carbon trails Astra and Opus. The hosts note OpenAI and Anthropic hold back their best internal models, so the public gap understates the real one.

47:25 Public vs. Private Tasks and What’s Next

Publishing every task lets experts verify quality, but it also lets labs hill climb by buying lookalike data. GDPx releases one task and rollout per partner, documents each partner’s methodology in depth, and explains its QA bar. Next, the team hopes to see GDPx on major model release cards, add partners and tasks, run every model at every reasoning level, and ship new versions as v1 saturates. They are hiring, and partners or labs can reach them at the GDPx email.

Notable Quotes

“A lot of people make their tasks harder by making it unfair rather than actually making the task like difficult.”
Max Kan, on why most popular AI benchmarks are broken

“The first trillion dollars of AI revenue are going to be highly concentrated within a few domains that can actually absorb sort of increasingly smarter frontier intelligence and earn high ROI.”
Max Kan, on the Tokenomics team thesis behind the GDPx domain list

“Look, using the models yourself for something that you think you’re pushing the frontier on absolutely gives you insight into which models are good and which models are bad, but you have to actually have that work to do.”
Jordan Nanos, after spotting Muse Wonder as a top cyber model before the benchmark did

“If you were a real accounting associate and then you gave your boss a workbook where like half the numbers are right and half of them are wrong. They’re not gonna be like, man like great job, little Jimmy, like you got half of it right.”
Max Kan, defending binary grading for finance tasks

“Good software engineers are more valuable today than they’ve ever been.”
Max Kan, on the Goldilocks zone for human plus AI work

“If you make it public it just becomes like way too easy to hill climb on the benchmark.”
Max Kan, on why most GDPx tasks stay private

Watch the full conversation here.