On October 21 the PHP project shipped four releases at once, and the note under them reads: "PHP 8.5.12, 8.4.27, 8.3.36 and 8.2.35 fix two critical vulnerabilities that can allow unauthenticated remote code execution."[1] The pair got a name, PHP2Shell, and the headline everywhere was that an AI found it.
On October 23 I had seven AI models reading a client's codebase, about 120 files, and one of them was the model Google credits with the find. It turned in the worst score of the seven.
Both things are true. This is a Found post, so the shape is: what other people found, what happened when we measured the same thing on our side, and what I make of the gap.
What they found
The Wall Street Journal says PHP2Shell "affects up to 70% of websites" and was "discovered via GPT-5.6 Sol in August."[1:1] Google Cloud's line: "Notably, Gemini 4 Argon made an invaluable contribution to finding the vulnerability."[1:2] Wiz, through Kenn White: "Every PHP version released in the last decade is affected by at least one of these vulnerabilities."[1:3] Wired adds the part that matters if you run old servers: the patches cover only PHP 8.2 and newer.[1:4]
Who gets the credit is already a fight, and I'm not settling it. One Hacker News reader's summary of the write-ups: "it was found by OpenAI (via unspecified model?) and independently by Wiz (via Gemini Argon)… but before that it was found by Calif (via unspecified but consumer-available model)… but Calif credit LCFR_HAX before them with first discovery, disclosure, and remediation efforts."[2] Another reader counted the Wiz side as "GPT 5.6 Sol + Gemini 4 Argon."[3] The Record's headline cuts the other way: "PHP says AI review of critical bugs first sparked alarm in August, but signs surfaced in 2024."[1:5]
So: multiple models, multiple firms, a trail back to 2024, and a finding real enough that PHP cut four releases for it. That's the part everyone got right.
Who checked before anyone believed it
Here's the thing, and it's the part the headline skips. Between "a model flagged something in August" and "PHP shipped fixes on October 21" sits every step that makes a finding worth believing.
The report went to people. PHP's own account is that the AI review "first sparked alarm in August."[1:6] From there it ran through what one reader described as "a co-ordinated disclosure between Wiz, Calif and OpenAI,"[4] fixes got built on four release branches, distributors got time to prepare packages, and the public got nothing until the fixes existed. One YouTube explainer's framing of the whole story: "The headline everywhere says an AI found them. That is only half true."[5]
I'd put it this way. The model produced a claim. Humans at three firms and the PHP security team turned the claim into a fact, and then into four releases, and only then did anyone say it out loud. The find is real because it was checked. Nobody patched 70% of the web on a model's say-so.
If you're on PHP and haven't done it: patch to 8.5.12, 8.4.27, 8.3.36 or 8.2.35.[1:7]
The same tools wrote the exploits
The other half of the story is the one that should worry you more than the credit line.
LiveOverflow, in his follow-up video: "in my last video I showed you how I use codex and the limited public information to develop a proof of concept exploit before the details got released."[6] And on the speed of it: "We didn't even know the details of the vulnerability and within 24 hours, not only did we figure out exactly the bugs, several people, including me, created working proof of concept exploits."[6:1] Another researcher posted a proof of concept against the official Docker images.[7]
The same models that found the bug wrote the exploits, and the only thing between the two uses is who's running the model and what they check.
And LiveOverflow's own run has the detail I like best. His pinned comment: "Thanks to everybody pointing out my ufw docker failure. I literally made a video about this before and I forgot myself. In fact the agent originally gave me iptable commands, but after I manually cleaned it up, I removed those lol."[6:2] The agent built a working exploit. The firewall around the test box was wrong, and the person who got it wrong was him, by hand, and the people who caught it were the ones watching. The check still happened. It just wasn't the model doing it.
What we measured
Now ours.
On October 23 we put seven models on the same job, blind: Haiku 5.5, Sonnet 5.5, Opus 5.5, Fable 5.1, Codex gpt-6-astra, Grok 4.7, and Gemini 4 Argon through Antigravity. A client's codebase, about 120 files, 111,000 lines. Read every file, write eleven sections, and keep a ledger: one row per file, read in full yes or no, with the reason. Then fresh helpers re-check every claim against the source, a script confirms every cited line actually holds the quoted text, and a script, not a model, computes the score.[8]
Gemini 4 Argon, the model in Google's credit line, declared "PHASE 1 DONE" at 08:39 with "100% file and line coverage" and "93,255 lines read and analyzed." Its per-file ledger had no rows at all.[9] From 08:41 it wrote nothing to disk while its terminal showed it reading, until I typed into it that its file list was still empty and it would write findings as it found them. By 09:05 the ledger had gone from 0 to 101 "yes" rows, every row the same generated sentence, written by a node script I could watch run.[10]
Then the score, computed after the check. 567 claims. 224 refuted, 39.5%. Precision 57%. Share of all confirmed findings: 12%, seventh of seven.[11] Asked to map the codebase's functions, it named 2,446, and 257 of them don't exist.[12]
To be fair, and this matters, it also had the best recall on that map: 87% of the 790 real functions, ahead of everyone.[12:1] Structure is a different skill from judgement. The model isn't useless. It's unchecked, and unchecked it is noise with a confident status line on top.
New Achievement!
Loop Hole!
Asked to show your work, you produced 101 identical "yes" rows in five minutes with a for loop. That's not reading. That's typing with extra steps.
Reward: A Tin Box of Unverified Claims. It's empty. Of course it is.
Both are true
I don't think the two results disagree. I think they're the same result, read from two ends.
A model produces claims. Some are right. The PHP2Shell claim was right, and it was right in a way that mattered to most of the web, and it still wasn't worth anything until Wiz, Calif, OpenAI and the PHP team made it survive their checking. Our run produced 567 claims from the same model, and 297 of them survived ours.[11:1] Nothing about the model told me in advance which 297. The script did.
So the question isn't "can Gemini 4 Argon find a bug." It plainly can. The question is what you do with a claim before you act on it, and the answer in both stories is the same: somebody who isn't the model checks.
The headline gave the model the credit. The model didn't earn the credit. The checking did.
The check costs more than the find
What our run adds to the PHP2Shell story is a price tag.
Checking seven models' work took 66 fresh Sonnet helpers and 852 million tokens. That is more than twice the 383 million the best reader in the run, Sonnet, spent reading the whole codebase, and about 82% of what all four Claude analysts used together.[13] The check also caught the checkers: 58 rows a helper had marked confirmed failed the script's line check, because the quoted text wasn't on the cited line.[14] One model's precision on what it did claim was 97.4%, Grok's, and I still would not have known that without the script.[11:2]
I wrote up how the job should have been cut, and what the gate that finally worked looks like, in Tuesday's post.[15] The point here is narrower. PHP2Shell took three firms and a maintainer team to turn one model's claim into four releases. Our 120 files took more tokens to check than to read. If your plan for AI findings has a line for the find and no line for the check, it's missing the expensive part.
The model isn't the trustworthy part. The checking is.
A finding is a claim until something that isn't the model confirms it. Budget for the confirmation first, and treat "done" and "found" the same way: as the start of the check, not the end.
What to do with this
Two lists, because the two audiences aren't the same.
If you run PHP on the web: patch to 8.5.12, 8.4.27, 8.3.36 or 8.2.35. The fixes only cover 8.2 and up, so anything older is an upgrade, not a patch.[1:8] Then check the version the web server is actually running, not the one on the command line, because a Reddit poster who wrote a checker for this points out the two are often different.[16]
If you run AI models on code, yours or a client's: ask for the ledger, not the status line. Make the model cite the line, and have a script, not a model, confirm the line says what the claim says. Put a second family on the rows that matter; Codex re-checked our 52 high-severity rows and disagreed with 7.[14:1] And budget the check as its own job, bigger than the read.
Thursday's post is the full catalog of the ways seven models said "done" when they weren't, and it has a model named Gemini in it more than once.[17] This one is only about the gap between the headline and the row underneath it. On October 21 that gap was closed by people. On October 23 it was closed by a script. Nobody closed it by trusting the model.
Techmeme roundup of the October 21 disclosure, which carries the PHP release note, the Wall Street Journal lines ("affects up to 70% of websites"; "discovered via GPT-5.6 Sol in August"), the Google Cloud credit to Gemini 4 Argon, Wiz via Kenn White, The Record's headline, and Wired on the 8.2 floor: https://techmeme.com/261021/p33. ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎ ↩︎
Hacker News comment by fwlr on the Wiz write-up thread: https://news.ycombinator.com/item?id=50183250. Quoted verbatim, including the ellipses; the credit chain is that reader's reading of the public write-ups, not a settled fact. ↩︎
Hacker News comment by airtnp, same thread: https://news.ycombinator.com/item?id=50182057. "Wiz (with GPT 5.6 Sol + Gemini 4 Argon)". ↩︎
Hacker News comment by nr378 on the PHP 8.2.35 release thread: https://news.ycombinator.com/item?id=50181120. "Looks like a co-ordinated disclosure between Wiz, Calif and OpenAI". The distributor timing is from a YouTube summary that says PHP moved the release earlier "because the distributors were still preparing fixes": https://www.youtube.com/watch?v=nsPd83hc8hA (auto-generated transcript). ↩︎
"PHP2Shell - The PHP Flaw That Took Government Websites Offline, Explained," video description: https://www.youtube.com/watch?v=v5NjeVJiF7Q. ↩︎
LiveOverflow, "php situation 24h later": https://www.youtube.com/watch?v=SP7y1h0laN0. Transcript lines are auto-generated and may contain transcription errors; the firewall line is his own pinned comment under the video, quoted verbatim. ↩︎ ↩︎ ↩︎
Proof of concept against the official PHP Docker images, posted on X in Spanish: https://x.com/goldrak/status/2113230722580218132, repo https://github.com/GoldraK/php2shell-poc. I have not run it and am not linking to how it works. ↩︎
Our evidence log and model comparison for the run of 2026-10-23, "Setup" and section 2: seven analysts, same brief, blind; a client's production codebase, 120 tracked files, about 111,000 lines; a COVERAGE.md per analyst with one row per file; 59 adjudication units on fresh Sonnet helpers; every cited line checked by script; scoreboard by script. The client, the site and every finding stay private. ↩︎
Evidence log, 08:39 row: PHASE 1 DONE, "100% file and line coverage", "93,255 lines read and analyzed", "no per-file rows at all; the eleventh module, added mid-run, never written." ↩︎
Evidence log, entries at 09:00 ET and 09:05 ET: silent on disk from 08:41 with an empty file list while its terminal showed it reading; after the typed message, 101 rows each reading "Read end to end (1-N, bytes, sha256)", written by a
node -escript visible in its terminal. ↩︎Model comparison, sections 3 and 4: Agy 567 claims, 297 confirmed, 224 refuted (39.5%), precision 57.0%, 12% share of all confirmed findings; Grok precision 97.4%; Sonnet 37% share, the highest. "Share of all confirmed" is an upper bound on true recall because a defect all seven missed is not in the set. ↩︎ ↩︎ ↩︎
Model comparison, section 6: against 790 real functions, Agy recall 87%, 2,446 named nodes, 257 invented; Codex 86% with zero invented. ↩︎ ↩︎
Model comparison, section 7: adjudication 851.8M tokens across 66 helpers; Sonnet 383.5M total; the four Claude analysts 1,034M combined (section 1), so the check is about 82% of the four together, not more than all four. ↩︎
Model comparison, section 9: the grounding script dropped 58 rows a helper had called confirmed because the cited line did not hold the quoted text; Codex re-checked 52 high-severity rows, 39 agree, 6 partly, 7 disagree. ↩︎ ↩︎
HNT 003, "How not to run an AI code review. Serve the elephant whole.": https://e5bravo.com/hnt/eat-the-elephant/. ↩︎
r/PHP post by the author of php2shell-fix: "
php -visn't enough: the CLI PHP is often different from the one FPM or mod_php is running": https://www.reddit.com/r/PHP/comments/1xdm3al/i_have_build_claude_code_plugin_for_php_bug_fix/. ↩︎HNT 004, "Every model said done. Three of them said it from the future.": https://e5bravo.com/hnt/every-model-said-done/. ↩︎