GitHub and Microsoft's AI Scanner Cut 9,791 Findings to 99 Verified Reports: 55 Vulnerabilities Fixed in Caddy, ImageMagick, Express, Netty, FastMCP and AutoGPT

A GitHub–Microsoft pilot used the MDASH AI scanner, a triage agent and human verification to send six open-source projects 99 private, patch-ready reports, and 55 flaws are now fixed.
GitHub and Microsoft have published the results of a pilot that used an AI scanner on six widely used open-source projects. Unlike most AI bug-hunting campaigns, it did not leave maintainers with a pile of raw findings. Of 9,791 candidate issues, only 99 verified reports reached the maintainers, each with a proof of concept and a suggested fix. By 15 November, 55 of those flaws had been fixed and 14 fixes had already shipped in releases.
The six projects were AutoGPT, Caddy, Express, FastMCP, ImageMagick and Netty. Together they sit under a large share of modern web servers, Node.js applications, Java network stacks, image pipelines and AI agent tooling.
What Happened
GitHub announced the results on 17 November, the same day Microsoft introduced MAI Cyber 2 Flash, a new security model, at its Ignite conference. GitHub's title sums up the pilot's message: maintainers don't need more AI bug reports, they need fixes.
All six projects are alumni of the GitHub Secure Open Source Fund and agreed to be scanned. Maintainers kept full control over what to fix, how and when. GitHub says the first fixes shipped about a week after the first scan.
How the Pipeline Worked
The pilot ran in five stages, each narrowing the output of the one before:
- Find: Codename MDASH, Microsoft Security's multi-model agentic scanning harness, scanned each codebase with the earlier MAI Cyber model and proposed a patch for each finding.
- Filter: A GitHub Security Lab triage agent, built on tailored Copilot skills, removed duplicates, checked validity and pushed higher-impact bugs to the top.
- Verify: Seven researchers from Microsoft's FORGE Lab checked every remaining finding by hand, which took about two days per project.
- Report privately: Verified findings went through GitHub private vulnerability reporting (PVR) with a proof of concept and a suggested fix, never as public issues.
- Fix: Maintainers decided what to fix, how and when to release.
That works out to roughly one report for every 100 raw findings. GitHub's point is that triage should be done by whoever sends the report, not by the volunteer who receives it.
What the Maintainers Saw
GitHub shared each project's experience:
- Caddy received 22 reports and fixed all of them within a week or less. Maintainers said the scan found obscure bugs that earlier reports had missed.
- ImageMagick, run by a two-person team, also received 22 reports. One looked like a known class of bug but wasn't, and the report pointed to the exact lines. A fix was pushed in 15 minutes. The maintainers say a typical advisory takes 30 to 60 minutes to read and fix.
- AutoGPT had shipped a security check that turned out to be broken. The report arrived three days later and identified the faulty line. Maintainers wrote a fix within a day that went further than the suggested patch, and it reached production in under a week.
- Express maintainers said the verified reports "cut through the noise."
- FastMCP, a framework for building Model Context Protocol servers, said the scan found issues that earlier AI scanners had not.
- Netty's maintainer said finding vulnerabilities is no longer the bottleneck. Fixing them is.
GitHub did not publish the individual vulnerabilities, CVE IDs or advisory IDs behind the 55 fixes. It also did not break down their severity.
Why It Matters
Open-source maintainers are dealing with a flood of AI-generated security reports. Many are duplicates, misreadings of the threat model, or "vulnerabilities" that cannot actually be exploited. Every one still takes a volunteer's time to read and reject.
The pilot argues that the fix is a higher bar for the sender, not a ban on AI. GitHub proposes a standard for anyone reporting with AI tools:
- Check every finding against what the project already knows, its threat model and its security policy.
- Report privately, never in a public issue.
- Include a suggested fix and let maintainers set the pace.
The post quotes Log4j maintainer Christian Grobmeier: "AI defending open source has to be better than the AI attacking it." That matters because attackers are using the same capability. AI agents that read patches and write exploits are already being used in real intrusions. Bugs that defenders find and fix first are bugs attackers cannot weaponise.
The Caveats
GitHub is open about what doesn't scale yet:
- Manual verification is the bottleneck. Seven specialist researchers spending two days per project worked for six projects. It does not work for thousands.
- Low-quality AI reports still arrive every day from other sources.
- Fix delivery needs work. GitHub wants suggested fixes to flow directly through PVR, and maintainers need CI/CD testing on private forks to validate a patch before disclosure.
- Maintainers need cyber-capable models of their own to reproduce a proof of concept before and after a fix.
The sample is also small and self-selected. All six projects are well funded by open-source standards, had security training through the fund, and opted in. Results for a one-person project with no security process could look very different.
What Comes Next
GitHub says it will offer AI security scans to every alumni project of the Secure Open Source Fund. Findings will be checked before private reporting, and alumni also get GitHub Copilot Max. Lessons from the pilot will go into private vulnerability reporting and security advisories. The fund says it has supported 216 projects and more than 300 maintainers so far.
On the scanning side, Microsoft says MAI Cyber 2 Flash will be available only inside its security products, including MDASH. From mid-December it will replace the current model in MDASH's MAI profile for eligible customers. Microsoft reports a 92.3% task success rate on the CyberGym-E2E benchmark after excluding 92 tasks that overlapped with its training data. These are vendor-reported figures and have not been independently reproduced.
What To Do Now
- Maintainers: turn on private vulnerability reporting for your repositories so AI-assisted and human researchers have a private channel, and publish a security policy that states your threat model.
- Teams that depend on these projects: watch for new releases of Caddy, ImageMagick, Express, Netty, FastMCP and AutoGPT and their security advisories. With 14 of 55 fixes released by 15 November, more advisories are likely to follow as fixes ship.
- Anyone using AI scanners on someone else's code: follow the standard above. Verify the finding, report privately, include a fix, and don't send raw scanner output.
