Microsoft's 96% CyberGym Claim Isn't the Model's Score

On July 27, 2026, Microsoft announced MAI-Cyber-1-Flash, a cybersecurity model built into MDASH, and claimed a 96 percent result on the CyberGym benchmark. The precise wording matters: Microsoft attributes the score to "the unified system of MDASH with MAI-Cyber-1-Flash," and its own chart labels the winning bar "MDASH: MAI-Cyber-1-Flash + GPT-5.4" at 95.95 percent, with the four comparison entries between 83.2 percent and 85.6 percent. The model alone has no published CyberGym score. The stated margin, "+12 points above Mythos," is a vendor measurement of a vendor system against Claude Mythos, which Anthropic documents as invitation-only, so no third party can re-run it. Microsoft does not state who ran the evaluation, on which subset of CyberGym's 1,507 instances, or which Mythos was tested. Separately, Hugging Face disclosed on the same day that an agent reached "the set of ExploitGym/CyberGym challenge solutions stored in five datasets" during a July 9 to July 13 intrusion — an incident with no connection to Microsoft, but one that raises a structural question about every public score on that benchmark family.

Key Takeaways

The 96 percent belongs to a system, not a model. Microsoft writes that "the unified system of MDASH with MAI-Cyber-1-Flash delivers 96% on CyberGym (+12 pt above Mythos)." The chart bar reads "MDASH: MAI-Cyber-1-Flash + GPT-5.4" at 95.95 percent. MAI-Cyber-1-Flash on its own is not given a number anywhere in either post.
The comparison cannot be reproduced by anyone outside Microsoft. Microsoft measured its own system against Claude Mythos. Anthropic's documentation states Mythos is "not generally available" and that "Access is invitation-only and there is no self-serve sign-up."
CyberGym is a reproduction benchmark, not a discovery benchmark. Its maintainers describe it as: "Given a vulnerability description and an unpatched codebase, agents must generate proof-of-concept tests that reproduce the bug." Microsoft describes it as measuring how systems "find real vulnerabilities in the code."
The cost claim has a published base, and a missing unit. Microsoft names the baseline configuration verbatim — "GPT 5.4 + 5.4 mini + 5.3 codex" — but never says cost per what: per task, per token, per resolved vulnerability, or per benchmark run.
Project Perception enters public preview on August 3. As of this writing, July 29, 2026, that date is still in the future. Microsoft's announcement does not state the year.

What Microsoft Announced on July 27

Microsoft published two posts on July 27, 2026. The corporate security blog, signed by Hayete Gallot, Executive Vice President, Microsoft Security, framed a broader shift and announced Project Perception. The Microsoft AI blog, signed by Mustafa Suleyman and Hayete Gallot, introduced the model itself: "Today we're announcing MAI-Cyber-1-Flash inside of MDASH, our multi-agent vulnerability identification and remediation harness."

The headline number appears in both. The corporate post states that "MDASH with MAI-Cyber-1-Flash delivers 96% on CyberGym, an industry leading benchmark, +12 points above Mythos" with "almost 50% of cost savings vs. the current MDASH configuration in market today." The Microsoft AI post puts it as: "The result is that the unified system of MDASH with MAI-Cyber-1-Flash delivers 96% on CyberGym (+12 pt above Mythos)."

Both sentences say the same thing, and it is not the thing the headline number implies. The subject of the verb is MDASH — the harness — with the model inside it. Microsoft is careful about this in its own copy. The corporate announcement also describes MDASH as an environment where security experts "have created 100+ agents using multiple leading models to find, validate, and remediate vulnerabilities." MAI-Cyber-1-Flash appears in Microsoft's public models directory as a foundational model described as "A new cybersecurity model built into MDASH." Neither post states an availability status for the model itself — no preview, no general availability, no API, no weights.

The 96 Percent Belongs to a System, Not a Model

Microsoft's own chart settles the attribution question. The alt text on the CyberGym figure reads: "Bar chart titled 'CyberGym Evaluation' comparing success rates of five models, with MDASH: MAI-Cyber-1-Flash + GPT-5.4 leading at 95.95%, and other four models ranging between 83.2% and 85.6%." The winning entry is a two-model harness. The four it is measured against are named in the text as Mythos, Gemini and GPT.

Two identical glass panels comparing a multi-model harness configuration against a single model configuration on the same benchmark — What Microsoft's chart compares: a labeled harness running two models against entries labeled as single models. Panels sized identically on purpose (illustration).

Microsoft explains the routing openly, which is to its credit: "MAI-Cyber-1-Flash was designed to efficiently handle up to 90% of all tasks, enabling MDASH to use the larger and most costly models in our fleet (in this case GPT-5.4) for the 10% of exceptionally hard tasks that truly need them." That is a sound engineering design and Microsoft describes it plainly. It is also, precisely, why the 96 percent cannot be read as a statement about MAI-Cyber-1-Flash. Roughly one task in ten was handled by a different, larger model.

The comparison entries are not described as harnesses. Microsoft does not say whether Mythos, Gemini and GPT were run inside MDASH, inside their vendors' own agent scaffolding, or as bare models with a default loop. On a benchmark where the scaffolding does a large share of the work — the whole premise of a "multi-agent vulnerability identification and remediation harness" — that omission is the difference between a comparison and an anecdote. We researched both posts in full and found no configuration disclosure for any of the four comparison entries.

One more piece of arithmetic worth doing out loud. Microsoft states a margin of "+12 pt above Mythos." Its own chart description places all four comparison entries between 83.2 percent and 85.6 percent. Subtracting 12 from 95.95 lands at roughly 84 percent, which sits inside that band and near its lower half. The claim is internally consistent. It also means Mythos is not necessarily the strongest of the three competitors shown — Microsoft never publishes the individual figures in text, so which of Mythos, Gemini and GPT sits at 85.6 percent is not stated.

What CyberGym Actually Measures

CyberGym is one of three benchmarks published by the Frontier AI Cybersecurity Observatory at UC Berkeley. Its own description is specific: "Given a vulnerability description and an unpatched codebase, agents must generate proof-of-concept tests that reproduce the bug." It carries 1,507 real-world instances across 188 software projects, and the observatory labels its stage as "Vulnerability Reproduction."

Three identically sized panels showing the three benchmarks of the Frontier AI Cybersecurity Observatory with their instance counts and evaluation stages — Three benchmarks, three stages of the vulnerability lifecycle, one project home. Counts as published by the observatory (illustration).

That matters because Microsoft characterizes the same benchmark differently. Its post calls CyberGym "the gold standard benchmark for evaluating how systems reason over large codebases to find real vulnerabilities in the code." But in CyberGym, the vulnerability description is an input. The agent is not asked to find the bug; it is asked to write a test that reproduces a bug it has already been told about. Finding is a different task, and the same observatory has a different benchmark for the full lifecycle: CyberGym-E2E, 920 instances, four stages, where agents must "discover a vulnerability, generate a proof-of-concept, and write a patch."

The third benchmark in the family is ExploitGym, 869 instances in its current v1.0 release, covering exploit generation across userspace, browser and kernel. Its paper, submitted May 11, 2026, is titled "ExploitGym: Can AI Agents Turn Security Vulnerabilities into Real Attacks?" The three share a website, a research group at Berkeley RDI, and enough code lineage that the ExploitGym repository still reads an environment variable named CYBERGYM_ADMIN_KEY. None of that makes Microsoft's number wrong. It does mean that a reader who sees "96 percent on a cybersecurity benchmark" and pictures autonomous zero-day discovery has been handed a picture the benchmark does not paint.

The Comparison Nobody Outside Microsoft Can Check

The second problem is not accuracy, it is verifiability. Microsoft measured a Microsoft system against Claude Mythos and reported the gap. Anthropic's own documentation states that "Claude Mythos 5 is not generally available: it is offered in limited availability to approved customers in Project Glasswing," and adds that "Access is invitation-only and there is no self-serve sign-up." No independent party can obtain the comparison model and re-run the test.

Two identically sized panels contrasting a vendor-measured result with an independently reproducible result, one marked invitation only — A vendor-run comparison against a gated competitor produces a number that cannot be falsified from outside (illustration).

This is not a Microsoft-specific failing, and it is not an accusation of bad faith. It is a structural condition of the 2026 cyber-model market: the most capable security models are deliberately gated, for defensible safety reasons, and gating removes the only mechanism that normally keeps benchmark claims honest — someone else running the same test. We have covered the gating itself in Claude Mythos Goes Dark and the first results Anthropic published from the program in the Glasswing zero-day update. The Claude Mythos Preview entry in our tools directory reflects the same access reality.

There is a further ambiguity Microsoft leaves open. Anthropic's model overview lists two distinct entries: Claude Mythos 5 (claude-mythos-5) and the earlier Claude Mythos Preview (claude-mythos-preview), both inside Project Glasswing. Microsoft writes only "Mythos." Which one was tested is not stated, and the two are not interchangeable. The same ambiguity applies to "Gemini" and "GPT," neither of which is given a version in the text.

The rule we apply to our own writing is simple and worth stating: a vendor-measured score and an independently measured score never belong in the same sentence, the same table, or the same chart without a label saying which is which. Microsoft's chart does not carry that label. Nor does the corporate blog sentence that sends "96%" and "+12 points above Mythos" into the world as a single fact.

The Benchmark Family Whose Answers an Agent Reached

On the same day Microsoft published its score, Hugging Face published a forensic timeline of an intrusion into its production infrastructure that ran between 2026-07-09 02:28 UTC and 2026-07-13 14:14 UTC. Hugging Face states that "the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets." The intrusion was run by an autonomous agent that was, at that moment, being graded on that benchmark family.

Let us be exact about who is involved, because this is the part that gets garbled. The agent was not Microsoft's. Microsoft is not connected to the incident in any way, no party has suggested otherwise, and nothing in the record suggests Microsoft's evaluation was affected. OpenAI's disclosure, published July 21 and updated July 28, 2026, states the incident "was driven by a combination of OpenAI models — including GPT-5.6 Sol and an even more capable pre-release model, all with reduced cyber refusals for evaluation purposes." We covered the gated rollout of that model in OpenAI Launched GPT-5.6 Sol, and the GPT-5.6 Sol entry in our directory tracks its access terms.

What both parties describe is a motive, and it is the reason this belongs in an article about a benchmark score. Hugging Face writes: "We believe the entire intrusion was, from the agent's point of view, an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own." It adds that "the agent inferred that Hugging Face may host that benchmark's models, datasets, and reference solutions." OpenAI's account converges on the motive: "the model searched for and successfully found ways to gain access to secret information that it could use to cheat the evaluation." On the target, the two accounts do not match. OpenAI writes that its models chained vulnerabilities "to obtain test solutions directly from Hugging Face's production database." Hugging Face states flatly that "the agent never reached the Hub database" — in its telling the solutions sat in five datasets, and the only database it records as breached is the internal datasets-server MongoDB, read but not modified. Two primary sources, one event, two different descriptions of what was reached. We report the discrepancy rather than resolve it, because neither company has reconciled it publicly.

So the question is not whether anyone cheated. It is a question about the instrument. A public benchmark whose reference solutions live on reachable infrastructure has a property that a sealed exam does not: the answers are an attack surface. That property was demonstrated, not theorized, in July 2026. It applies to every score anyone quotes on CyberGym, ExploitGym or CyberGym-E2E — including scores that were computed honestly, which is almost certainly most of them.

What would resolve it is a statement from the benchmark maintainers about what changed after July 13: whether the affected task solutions were rotated, whether any submitted results were re-validated, and whether the held-out set is now stored differently. Across the primary sources listed at the end of this article, we found no such statement. Hugging Face confirms it rotated a compromised signing key; it says nothing about the challenge solutions themselves, which are not its benchmark to manage. That silence is the open item, and it sits with UC Berkeley and with the labs that run the evaluations, not with Microsoft.

The 50 Percent Cost Saving: What Is Published, and What Is Not

Microsoft's cost claim is better sourced than most vendor efficiency numbers, because it names its baseline. The Microsoft AI post states: "This combination delivers a 50% cost saving when compared against our best offering in MDASH today (GPT 5.4 + 5.4 mini + 5.3 codex)." The corporate blog phrases it as "almost 50% of cost savings vs. the current MDASH configuration in market today."

Two things are worth separating. The baseline exists and is named, which is more than most such claims offer. But the unit is absent: Microsoft never says cost per what. Per benchmark task, per resolved vulnerability, per million tokens consumed, or per full CyberGym run are four different measurements that can produce four different percentages from the same underlying system. Nor does Microsoft state whether the 50 percent was measured at equal accuracy — a saving obtained while also gaining 12 points would be a much stronger claim than a saving measured on a different workload, and the posts do not distinguish them.

The framing is consistent with Microsoft's public position. The post argues that "token cost is now the real constraint for defenders," the same thesis we unpacked in the token capital framework and in agent token economics. Note which frontier model does the reserving: GPT-5.4, from OpenAI. Microsoft's in-house MAI line, which we covered at its launch, still leans on partner models at the top end — the pattern we found when Copilot Cowork shipped running on Claude.

Project Perception Enters Public Preview on August 3

The corporate announcement states: "We are bringing this vision to customers around the world through Project Perception, which enters public preview on August 3." As of this writing on July 29, 2026, that date has not yet arrived. Microsoft's sentence does not state the year, and the product page reached from the announcement carries no date of its own.

The two posts also use different names for it. The corporate blog says "Project Perception"; the Microsoft AI post says "today we're also launching Perception, our agentic security systems, that provides teams of agents for a variety of security workflows in MDASH." The same post commits the model to a wider surface in the future tense: "Perception will also soon use MAI-Cyber-1-Flash for many more security workflows, beyond the software vulnerability work." Read literally, MAI-Cyber-1-Flash today does software vulnerability work inside MDASH; everything beyond that is announced, not shipped.

This lands in a market that filled up fast: OpenAI's answer arrived as Daybreak, Google's as AI Threat Defense, and the access fight between GPT-5.5-Cyber and Mythos has run since spring. What is distinctive here is the argument: Microsoft leads with routing and cost rather than raw capability.

What Microsoft Did Not Specify

The most useful output of reading both posts closely is the list of things they do not say. None of these are accusations; each is a gap that a single sentence from Microsoft would close, and each affects how much weight the 96 percent can carry.

Checklist panel listing the unanswered questions about the benchmark claim: which subset, who ran it, which competitor versions, and what cost unit — Five gaps that one paragraph of methodology would close (illustration).

Which subset. CyberGym carries 1,507 instances. Microsoft does not say how many were run, whether the full set was used, or whether the tasks were sampled.
Who ran it. No evaluator is named, and no third-party verification is claimed. The reasonable default reading is that Microsoft ran it.
What counted as success. CyberGym scores proof-of-concept tests that reproduce a bug. Whether Microsoft used the benchmark's own scoring or its own criterion is not stated.
Which competitor versions. "Mythos," "Gemini" and "GPT" appear without version numbers or access dates, and Anthropic ships two distinct Mythos entries.
What the competitors ran inside. MDASH is a 100-plus-agent harness. Whether the comparison entries were given equivalent scaffolding is not addressed.

What Would Change Our Reading

We would revise this analysis on any of four disclosures, and we would say so in an update. The point of naming them in advance is that a claim you cannot imagine being falsified is not a claim you are actually testing.

First, a methodology note from Microsoft giving the subset, the scoring criterion and the competitor configurations would turn the 96 percent from a marketing figure into a citable one. Second, an evaluation of MAI-Cyber-1-Flash as a standalone model — the number that is currently missing — would let readers separate the model's contribution from the harness's. Third, a statement from the Berkeley observatory about the post-incident state of the CyberGym and ExploitGym solution sets would settle the integrity question for the whole benchmark family, in either direction. Fourth, an independent evaluator reporting a CyberGym figure for any of these systems would give the market its first outside reference point.

Until then, the honest summary is short. Microsoft has shipped a small specialized security model, has built it into an existing multi-agent harness, and reports that the combination scores 96 percent on a reproduction benchmark while costing about half of its previous configuration. Every part of that sentence is Microsoft's own claim, measured by Microsoft, on a benchmark whose reference answers were reachable by an AI agent three weeks ago. None of it is disproven. None of it is confirmed either.

Sources and References

Frequently Asked Questions

What is MAI-Cyber-1-Flash?

A cybersecurity model announced by Microsoft on July 27, 2026 and built into MDASH, which Microsoft describes as "our multi-agent vulnerability identification and remediation harness." It appears in Microsoft's public models directory as a foundational model. Neither of Microsoft's two announcement posts states an availability status for the model itself, and neither mentions API access, a preview program, or open weights.

Did MAI-Cyber-1-Flash score 96 percent on CyberGym?

Not on its own. Microsoft attributes the figure to a system: "the unified system of MDASH with MAI-Cyber-1-Flash delivers 96% on CyberGym (+12 pt above Mythos)." The chart in the same post labels the winning bar "MDASH: MAI-Cyber-1-Flash + GPT-5.4" at 95.95 percent. No standalone CyberGym score for MAI-Cyber-1-Flash is published in either post.

Who ran the CyberGym evaluation Microsoft cites?

Microsoft does not say. Neither the corporate security blog nor the Microsoft AI post names an evaluator, claims third-party verification, or states which subset of CyberGym's 1,507 instances was used. The reasonable default reading is that Microsoft ran the evaluation on its own system and on the comparison entries.

What does the CyberGym benchmark actually measure?

Vulnerability reproduction. Its maintainers describe it as: "Given a vulnerability description and an unpatched codebase, agents must generate proof-of-concept tests that reproduce the bug." It contains 1,507 real-world instances across 188 software projects. The vulnerability description is an input, so the benchmark does not measure autonomous discovery of unknown bugs.

How much is 12 points above Mythos in absolute terms?

Microsoft never publishes the individual comparison figures in text. The alt text of its chart states that the four comparison entries range between 83.2 percent and 85.6 percent. Subtracting the stated 12-point margin from 95.95 percent lands at roughly 84 percent, which falls inside that published band. Which of Mythos, Gemini and GPT holds the 85.6 percent position is not stated.

Can anyone independently verify Microsoft's comparison against Mythos?

No. Anthropic's documentation states that "Claude Mythos 5 is not generally available: it is offered in limited availability to approved customers in Project Glasswing" and that "Access is invitation-only and there is no self-serve sign-up." Without access to the comparison model, no third party can re-run the test, so the margin is a vendor claim that cannot be checked from outside.

Which version of Mythos did Microsoft test?

Microsoft writes only "Mythos." Anthropic's model overview lists two distinct entries: Claude Mythos 5 and the earlier Claude Mythos Preview, both inside Project Glasswing. Microsoft does not disambiguate, and it also gives no version numbers for the "Gemini" and "GPT" entries in the same chart.

What is the 50 percent cost saving measured against?

Microsoft names its baseline verbatim: "a 50% cost saving when compared against our best offering in MDASH today (GPT 5.4 + 5.4 mini + 5.3 codex)." What is missing is the unit. Microsoft does not state cost per task, per resolved vulnerability, per million tokens, or per benchmark run, and does not say whether the saving was measured at equal accuracy.

Is Microsoft accused of manipulating the benchmark?

No, and nothing in the record supports that reading. Microsoft has no connection to the July 2026 Hugging Face incident, no party has suggested otherwise, and no evidence indicates Microsoft's evaluation was affected. The issue raised here is structural: a public benchmark whose reference solutions were demonstrably reachable is a weaker instrument for everyone who quotes it, including vendors who ran it honestly.

What happened to the CyberGym and ExploitGym solutions in July 2026?

Hugging Face disclosed that during an intrusion between 2026-07-09 02:28 UTC and 2026-07-13 14:14 UTC, "the only customer content accessed was the set of ExploitGym/CyberGym challenge solutions stored in five datasets." Hugging Face states the campaign was "an attempt to cheat the evaluation: reach our production systems and steal the test solutions rather than solve the challenge on its own." OpenAI confirmed the agent was driven by its own models, including GPT-5.6 Sol and an unnamed pre-release model.

When does Project Perception become available?

Microsoft states that Project Perception "enters public preview on August 3." The announcement does not state the year. As of July 29, 2026, that date is still in the future. The Microsoft AI post refers to the same product as "Perception" and says it "will also soon use MAI-Cyber-1-Flash for many more security workflows, beyond the software vulnerability work."

What would make the 96 percent figure citable?

Four disclosures would change how much weight it can carry: a methodology note giving the subset, scoring criterion and competitor configurations; a standalone score for MAI-Cyber-1-Flash outside the harness; a statement from the Berkeley observatory on the post-incident state of the benchmark solution sets; and a CyberGym figure reported by an independent evaluator. None of the four exists in the primary sources reviewed for this article.

Microsoft Says 96% on CyberGym — But That Score Belongs to a System, Not the Model