The Verdict, Four Months Later

Every series needs a place to land. Fifteen posts ago I opened with a roadmap. This is the close.

The hands-on testing behind this series happened in March 2026, in the preview window before AWS Security Agent's March 31 general availability. I am publishing this in late July 2026, four months later. In that gap AWS kept shipping, the wider AI-pentesting market kept moving, and I kept thinking about what I'd actually tell someone who asked me, today, whether to use this thing. This post is that answer, in four parts: the numbers the series is built on, the verdict distilled to its essentials, what has changed since I did the testing, and an honest look at how AI pentesting compares to hiring a traditional pentest firm, backed by real sources rather than vibes.

The Numbers, One More Time

Before the opinions, the receipts. This series is built on four full pentest runs (an unauthenticated baseline, a credentialed run, a business-logic-only target, and an adversarial trap built to attack the agent back), nine design reviews (a clean run, a six-review format experiment across two documents and three file formats, and two prompt-injection attempts), and four code reviews across three separate applications (a hinted PR, a commentless control, a heavily obfuscated application, and a pull request built entirely out of lies). I downloaded and read 3,433 CloudTrail events across 36 API actions, tens of thousands of CloudWatch log events including roughly 2,260 reasoning blocks and 2,966 tool calls from a single pentest run alone, and 64,814 VPC Flow Log connections from one authenticated run. Every number in the fifteen posts before this one traces back to one of those artifacts, and all of it is still sitting in the companion repository if you want to check any of it yourself.

That is the standard I held the rest of this series to, and it is the standard I am holding this closing post to as well, including the two sections below that go beyond my own lab. Where a claim comes from my own testing, I say so. Where it comes from AWS's own published word, I cite the page. Where it comes from outside research for this post specifically, I name the source and say plainly when that source is a market estimate rather than a standard, or when I could not verify something to my own satisfaction.

The Honest Verdict, Distilled

Article 11 has the full breakdown with every table. Here is what actually matters, compressed to five sentences.

The agent is genuinely good at detection once it can see the target: 9 of 10, then 10 of 10 with one custom requirement, in code review; 18 of 19 against a codebase deliberately obfuscated with encoding and dynamic dispatch. It is comparatively weak at discovery: my unauthenticated crawler reached only 2 of 42 vulnerability endpoints, and everything downstream of that number was bounded by it, not by detection skill. Credentials are not optional: providing a login turned 5 findings into 13, a 160 percent increase, with all 6 Critical findings appearing only in the authenticated run. It is stochastic: the same configuration against the same application produced different validator outcomes, different discovery paths, and a different finding count across runs, and prompt injection and lying comments did not work against it in any of the four adversarial tests I ran, but that resistance is a property I measured across single runs, not a guarantee. And it cannot infer your authorization model from clean code on its own: IDOR only became detectable once I wrote a custom requirement that told the agent what "authorized" means for that application.

If you read nothing else in this series, that paragraph is the series.

What's Changed Since I Did This Testing

I want to be direct about the limits of this section the same way I was in the post about the agent's own attack surface: everything below comes from AWS's own public documentation and blog posts, not from hands-on testing, because I have not re-run my lab against the current GA service. Every date and claim below is sourced to an AWS domain.

May 12, 2026: full-repository code review entered preview. This was already the newest capability by the time I wrote the code-review posts. As of this writing it is still in preview; I found no AWS announcement of it reaching general availability through the date of this post (AWS What's New).

May 22, 2026: verification scripts for pentest findings. For each confirmed finding, the agent now generates a script with setup instructions and documented environment variables so a team can independently reproduce the vulnerability rather than take the report on faith (AWS What's New).

June 17, 2026 was a big day. Three things shipped together:

• Threat Modeling entered public preview. It generates STRIDE-based threat models automatically from design documents or source code, across all six STRIDE categories, at no additional cost during the preview (AWS What's New).
• A Kiro and Claude Code plugin shipped, along with MCP support and five new source-control integrations (GitLab.com, GitLab Self Managed, GitHub Enterprise, Bitbucket, and Confluence), letting developers build threat models, run code scans, and remediate findings without leaving their IDE. The same announcement introduced "simulated validation" for the code scanner: it now executes findings in an isolated environment and returns evidence of exploitability, rather than reporting a pattern match and leaving triage to you (AWS What's New).
• AWS introduced Continuum, and this is the one worth reading carefully rather than skimming past. Per AWS's own announcement, "AWS Security Agent penetration testing and code scanning are now available as Continuum penetration testing and Continuum code scanning (preview)." A broader Continuum offering for code vulnerabilities, which ingests findings from any tool, prioritizes them, validates exploitability in a sandbox, and routes fixes through your own review process, is in gated preview (AWS Security Blog, Continuum product page). I want to be careful about what I can and cannot conclude from that wording. It reads as an additive rebrand, two of AWS Security Agent's three capabilities are now also offered under a new name, not as a discontinuation. I found no announcement stating AWS Security Agent is being retired or renamed outright. If you are evaluating this product today, go in expecting to see both names in AWS's own materials, and confirm for yourself which one is current by the time you read this.

July 1, 2026: the region count grew from 6 to 9, adding Asia Pacific (Mumbai), Asia Pacific (Singapore), and South America (São Paulo) (AWS What's New), which the earlier setup post in this series already covers.

Two more data points from AWS's own GA announcement that I had not used elsewhere in this series and think are worth recording here: AWS reported a 92.5 percent success rate on CVE Bench v2.0, and cited customer results including HENNGE K.K. reporting more than 90 percent faster testing, Scout24 SE reporting the agent outperformed traditional DAST tooling, Bamboo Health reporting findings no other tool had surfaced, and LG CNS reporting over 50 percent faster testing and roughly 30 percent lower costs (AWS Security Blog, AWS Weekly Roundup, April 6, 2026). I have seen more dramatic numbers than that LG CNS figure reported in third-party press. I am deliberately using AWS's own, more conservative figure here, because that is the standard this whole series holds itself to.

AI Pentesting Versus Traditional Pentest Consultancy

This is the section I was most wary of writing, because it is the easiest place in this whole series to slide from data into marketing copy, in either direction. So here is what I actually found when I went looking, with sources, and here is where the trail runs cold.

Pricing does not compare cleanly, and I want to say that plainly before giving you numbers. No standards body publishes penetration testing rates; price is a market question, not a compliance one. Triangulating across several named firms' own published guidance, day rates in 2026 commonly run somewhere in the $1,200 to $3,000-plus range in the US, with a full web application engagement typically landing between $5,000 and $30,000 depending on scope, and larger red-team engagements running past $100,000 (Synack, pentest pricing guide). I went looking for a more authoritative anchor than a vendor's own pricing blog, and I want to tell you honestly that I did not find one. I initially had a cybersecurity workforce survey lined up to cite here for a compensation figure and a workforce-gap number, then checked the actual current report directly before publishing this and found it does not contain either figure, the organization that publishes it discontinued the workforce-gap metric entirely as of its most recent edition. So I am not citing it, and I am not replacing it with something else just to have a number in this paragraph. AWS's own worked pricing example for a comparable web application, roughly 24 task-hours at $50 each, works out to about $1,200 (AWS Security Agent pricing). That is a real, order-of-magnitude gap against the market-estimated traditional engagement range above, but it is not an apples-to-apples one: the two figures were not measured against the same depth or scope of testing, and I do not have a rigorous way to normalize them. Treat the gap as real and directional, not as a clean substitution price.

Every methodology standard that governs traditional pentesting predates AI pentesting entirely, and was written assuming a human tester. PTES defines seven phases from pre-engagement through reporting (PTES, the standard's own site). NIST SP 800-115 is the US government's own technical guide to security testing (NIST SP 800-115). CREST certifies both individual testers and firms, and is recognized by the UK's National Cyber Security Centre (CREST, Registered Penetration Tester). None of them currently define a role for an autonomous agent as the tester of record.

PCI DSS v4.0 makes this concrete, and it is the single most important fact in this section. PCI's Requirement 11.4 draws an explicit line between vulnerability scanning, which can be automated, and penetration testing, which its documentation and the compliance guidance built around it describe as requiring human-driven exploitation by a tester who is organizationally independent from the systems under test (PCI Security Standards Council, penetration testing guidance). I was not able to read that primary PCI Security Standards Council document character for character to quote it directly, its published guidance is image-heavy and did not extract cleanly, so I am not putting words in quotation marks here that I have not personally verified. But multiple compliance-focused sources describing the same requirement converge on the same reading, and I found no PCI SSC statement, no SOC 2 auditor guidance, and no other major compliance framework stating that an AI agent can satisfy a penetration testing requirement on its own as of today. If part of why you need a pentest is a compliance checkbox, that checkbox currently still needs a human name attached to it, on top of whatever automated testing you also run.

The category itself is real, not hype. Gartner published an "Innovation Insight for Agentic Application Security Testing" in June 2026, defining the category directly (Gartner document listing, the full text is paywalled, cited here via a named vendor's own discussion of being included in it). Gartner's separate Adversarial Exposure Validation market guide raised its own adoption forecast between its 2025 and 2026 editions, from 40 percent of organizations by 2027 to 60 percent (same paywall caveat, cited via Picus Security's and Hadrian's own discussion of the same report). I was not able to read Gartner's own paywalled documents directly, so both figures are one level removed from the primary source, and I am flagging that explicitly rather than presenting them as if I read Gartner's own text. Named, real, venture-backed competitors exist in this space, Horizon3.ai's NodeZero product being the one I found the most independently verifiable growth data for: roughly $186 million in disclosed funding and reported 102 percent year-over-year revenue growth as of March 2026 (Tracxn company profile). AWS entering this category as an integrated, first-party hyperscaler offering rather than a point-solution startup is itself a notable shift in the market, whatever you make of the product.

And here is where the trail runs genuinely cold, and I think that is worth saying plainly rather than papering over. I could not find a single rigorous, peer-reviewed, apples-to-apples benchmark anywhere that pits a named AI pentesting product against certified human testers on the same target, under the same time budget, scored the same way. What exists instead is softer, but still real. A Cobalt-sponsored survey reported by Dark Reading in June 2026 found that organizational confidence in AI-powered penetration testing actually fell, from 29 percent to 9 percent year over year, with 58 percent of respondents saying AI misses business logic or chained exploits, and most preferring a hybrid model with a human in the loop (Dark Reading). I'll note for balance that Cobalt is a pentest-as-a-service firm with a commercial stake in that hybrid narrative, so read the number knowing who funded the question. Separately, and more usefully, an academic preprint from May 2026 ran 400 trials of an LLM pentesting agent against one fixed vulnerable target and found significant variance in both exploitation success and attack strategy across runs, run to run, against the exact same target (arXiv:2605.30096, a preprint, not peer-reviewed). That is an independent, unrelated confirmation of the same non-determinism I measured myself in article 11 with my own two runs. I did not expect to find someone else's data point land that close to mine, and it is the single best piece of outside evidence in this whole section.

So where does that leave the comparison? Not at "AI replaces traditional pentesting." The standards that govern serious pentesting, including the one your compliance program probably cares about most, still assume a human. The market's own confidence in autonomous testing is trending down, not up, even as the category matures and gets real analyst attention. What the data supports is narrower and, I think, more honest: a fast, cheap, always-available complement that is strong at finding what it can see and weak at knowing where to look, sitting alongside human-led testing rather than replacing the human name on the compliance report. That is not a knock on the product. It is what four months of watching this space since my own hands-on testing, on top of that testing itself, actually shows.

Where I'd Use It, and Where I Wouldn't, Today

Given everything above, here is my own practical read, stated plainly rather than hedged into meaninglessness.

I would run it constantly, not occasionally. Task-hour pricing and on-demand availability make it cheap enough to run against every meaningful change, not save for an annual audit window. I would invest real effort in feeding it context before every run: a sitemap, an OpenAPI spec, or connected source code, because discovery, not detection, is the ceiling on what it finds. I would write custom security requirements for anything specific to how my application decides who is allowed to see what, because that is the one class of bug it will not find on its own no matter how many times I run it. I would run it more than once for anything I actually intend to trust the result of, because a single run is a sample, not a census. And I would not treat it as satisfying a compliance requirement that names a human tester, not because the product is weak, but because the standard itself has not caught up yet, and pretending otherwise is a compliance risk, not a security one.

Where to Go From Here

That closes the loop this series opened fifteen posts ago. The companion repository has everything: the vulnerable applications, the Terraform to stand them up, the raw pentest and code review data, and now this closing post's sources alongside it. If you run the same experiments and get a meaningfully different result, or if you find a mistake in anything across this whole series, including this closing post, I want to hear about it. I write this as an AWS Community Builder, which means I have every incentive to be generous to AWS's products and exactly zero excuse for being wrong about what they actually do. Every number in this series was checked against a log, a document, or an official source before it went in a post, and that standard does not end with this one.

Thank you for reading the whole thing.