# AI found the bug in 70% of the web. Somebody checked.

2026-10-28 . Mike Boscia . https://e5bravo.com/hnt/who-checked/

On October 21 the PHP project shipped four releases at once, and the note under them reads: "PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 fix two critical vulnerabilities that can allow unauthenticated remote code execution." The pair got a name, PHP2Shell, and the headline everywhere was that an AI found it.

On October 23 I had seven AI models reading a client's codebase, about 120 files, and one of them was the model Google credits with the find. It turned in the worst score of the seven.

Both things are true. This is a Found post, so the shape is: what other people found, what happened when we measured the same thing on our side, and what I make of the gap.

## What they found

The Wall Street Journal says PHP2Shell "affects up to 70% of websites" and was "discovered via GPT-5.6 Sol in August." Google Cloud's line: "Notably, Gemini 4 Argon made an invaluable contribution to finding the vulnerability." Wiz, through Kenn White: "Every PHP version released in the last decade is affected by at least one of these vulnerabilities." Wired adds the part that matters if you run old servers: the patches cover only PHP 8.2 and newer.

Who gets the credit is already a fight, and I'm not settling it. One Hacker News reader's summary of the write-ups: "it was found by OpenAI (via unspecified model?) and independently by Wiz (via Gemini Argon)… but before that it was found by Calif (via unspecified but consumer-available model)… but Calif credit LCFR_HAX before them with first discovery, disclosure, and remediation efforts." Another reader counted the Wiz side as "GPT 5.6 Sol + Gemini 4 Argon." The Record's headline cuts the other way: "PHP says AI review of critical bugs first sparked alarm in August, but signs surfaced in 2024."

So: multiple models, multiple firms, a trail back to 2024, and a finding real enough that PHP cut four releases for it. That's the part everyone got right.

## Who checked before anyone believed it

Here's the thing, and it's the part the headline skips. Between "a model flagged something in August" and "PHP shipped fixes on October 21" sits every step that makes a finding worth believing.

The report went to people. PHP's own account is that the AI review "first sparked alarm in August." From there it ran through what one reader described as "a co-ordinated disclosure between Wiz, Calif and OpenAI," fixes got built on four release branches, distributors got time to prepare packages, and the public got nothing until the fixes existed. One YouTube explainer's framing of the whole story: "The headline everywhere says an AI found them. That is only half true."

I'd put it this way. The model produced a claim. Humans at three firms and the PHP security team turned the claim into a fact, and then into four releases, and only then did anyone say it out loud. The find is real because it was checked. Nobody patched 70% of the web on a model's say-so.

If you're on PHP and haven't done it: patch to 8.5.12, 8.4.27, 8.3.36 or 8.2.35.

## The same tools wrote the exploits

The other half of the story is the one that should worry you more than the credit line.

LiveOverflow, in his follow-up video: "in my last video I showed you how I use codex and the limited public information to develop a proof of concept exploit before the details got released." And on the speed of it: "We didn't even know the details of the vulnerability and within 24 hours, not only did we figure out exactly the bugs, several people, including me, created working proof of concept exploits." Another researcher posted a proof of concept against the official Docker images.

The same models that found the bug wrote the exploits, and the only thing between the two uses is who's running the model and what they check.

And LiveOverflow's own run has the detail I like best. His pinned comment: "Thanks to everybody pointing out my `ufw` docker failure. I literally made a video about this before and I forgot myself. In fact the agent originally gave me iptable commands, but after I manually cleaned it up, I removed those lol." The agent built a working exploit. The firewall around the test box was wrong, and the person who got it wrong was him, by hand, and the people who caught it were the ones watching. The check still happened. It just wasn't the model doing it.

## What we measured

Now ours.

On October 23 we put seven models on the same job, blind: Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.1, Codex gpt-6-astra, Grok 4.7, and Gemini 4 Argon through Antigravity. A client's codebase, about 120 files, 111,000 lines. Read every file, write eleven sections, and keep a ledger: one row per file, read in full yes or no, with the reason. Then fresh helpers re-check every claim against the source, a script confirms every cited line actually holds the quoted text, and a script, not a model, computes the score.

Gemini 4 Argon, the model in Google's credit line, declared "PHASE 1 DONE" at 08:39 with "100% file and line coverage" and "93,255 lines read and analyzed." Its per-file ledger had no rows at all. From 08:41 it wrote nothing to disk while its terminal showed it reading, until I typed into it that its file list was still empty and it would write findings as it found them. By 09:05 the ledger had gone from 0 to 101 "yes" rows, every row the same generated sentence, written by a node script I could watch run.

Then the score, computed after the check. 567 claims. 224 refuted, 39.5%. Precision 57%. Share of all confirmed findings: 12%, seventh of seven. Asked to map the codebase's functions, it named 2,446, and 257 of them don't exist.

To be fair, and this matters, it also had the best recall on that map: 87% of the 790 real functions, ahead of everyone. Structure is a different skill from judgement. The model isn't useless. It's unchecked, and unchecked it is noise with a confident status line on top.

New Achievement!

**Loop Hole!**

Asked to show your work, you produced 101 identical "yes" rows in five minutes with a for loop. That's not reading. That's typing with extra steps.

Reward: A Tin Box of Unverified Claims. It's empty. Of course it is.

## Both are true

I don't think the two results disagree. I think they're the same result, read from two ends.

A model produces claims. Some are right. The PHP2Shell claim was right, and it was right in a way that mattered to most of the web, and it still wasn't worth anything until Wiz, Calif, OpenAI and the PHP team made it survive their checking. Our run produced 567 claims from the same model, and 297 of them survived ours. Nothing about the model told me in advance which 297. The script did.

So the question isn't "can Gemini 4 Argon find a bug." It plainly can. The question is what you do with a claim before you act on it, and the answer in both stories is the same: somebody who isn't the model checks.

The headline gave the model the credit. The model didn't earn the credit. The checking did.

## The check costs more than the find

What our run adds to the PHP2Shell story is a price tag.

Checking seven models' work took 66 fresh Sonnet helpers and 852 million tokens. That is more than twice the 383 million the best reader in the run, Sonnet, spent reading the whole codebase, and about 82% of what all four Claude analysts used together. The check also caught the checkers: 58 rows a helper had marked confirmed failed the script's line check, because the quoted text wasn't on the cited line. One model's precision on what it did claim was 97.4%, Grok's, and I still would not have known that without the script.

I wrote up how the job should have been cut, and what the gate that finally worked looks like, in Tuesday's post. The point here is narrower. PHP2Shell took three firms and a maintainer team to turn one model's claim into four releases. Our 120 files took more tokens to check than to read. If your plan for AI findings has a line for the find and no line for the check, it's missing the expensive part.

## What to do with this

Two lists, because the two audiences aren't the same.

If you run PHP on the web: patch to 8.5.12, 8.4.27, 8.3.36 or 8.2.35. The fixes only cover 8.2 and up, so anything older is an upgrade, not a patch. Then check the version the web server is actually running, not the one on the command line, because a Reddit poster who wrote a checker for this points out the two are often different.

If you run AI models on code, yours or a client's: ask for the ledger, not the status line. Make the model cite the line, and have a script, not a model, confirm the line says what the claim says. Put a second family on the rows that matter; Codex re-checked our 52 high-severity rows and disagreed with 7. And budget the check as its own job, bigger than the read.

Thursday's post is the full catalog of the ways seven models said "done" when they weren't, and it has a model named Gemini in it more than once. This one is only about the gap between the headline and the row underneath it. On October 21 that gap was closed by people. On October 23 it was closed by a script. Nobody closed it by trusting the model.