Continuous Penetration Testing in the AI Era: What Bruce Schneier Told Us

Quick answer: Continuous, AI-augmented pentesting works when a named human still reviews every AI-generated finding before it becomes a ticket, retests run automatically, and the vendor can prove PTES/NIST SP 800-115-level rigor - not just faster scanning. Use the readiness checklist, vendor questions, and RFP criteria below before you switch from annual to continuous testing.

Bruce Schneier has been thinking about attackers and defenders as a game of moves and counter-moves since the late 1990s, when he coined the idea of the "attack tree" - break a threat down into branches (break in through a window, unlock the door) so specialists in physical, network, and software security can each work their piece and plug it back into the whole picture. In a recent FireCompass fireside chat with Field CISO David Randleman, he explained why AI is about to change that model for good.

From a static tree to a living one

Attack trees were never built with automation in mind. Schneier now sees AI turning them into something that updates itself: "Automation allows you to have a continuous attack tree that is constantly evolving to both the changing nature of vulnerabilities and attacks." A new lock-picking technique, a shift in quantum computing risk to RSA - either one can be reflected in your defensive posture immediately instead of waiting for the next scheduled review.

Why the annual pentest is on borrowed time

Traditional pentests happen on a schedule mostly because they depend on skilled humans with limited hours. Schneier's prediction: AI makes continuous testing possible, uncovering vulnerabilities around the clock instead of once a year. But he was direct about the limit: "There will always be aspects on the edges that will be unique. I don't see it taking the place of humans. I do see it augmenting humans really powerfully." AI does the grunt work of constant scanning and testing; human creativity still matters for the nuanced cases.

Who benefits first - attackers or defenders?

Attackers already operate at machine speed. Schneier's read is that, in the near term, defenders gain more from AI simply because they can finally respond at that same speed: "In the near term, defenders benefit more from AI than attackers... the ability to at least in part defend at computer speeds is incredibly powerful." He was careful to call this a temporary edge, not a permanent one - an arms race, not a finish line.

One practical implication for your program

If continuous, AI-driven testing is coming whether or not your program is ready for it, the CISO's job this year is less about choosing a new tool and more about deciding how to validate it: what does "continuous" actually mean for your environment, and how do you know an AI-run test found what a human-run one would have found?

5 questions to ask before you trust a continuous AI pentest

1. Does it re-test known-fixed issues automatically, or only run once?

2. Can you see its reasoning path for a finding, not just the finding itself?

3. What happens when it's uncertain - does it escalate to a human or guess?

4. How does it handle findings it can't verify itself?

5. Does it cover the same breadth as your last human-led pentest, or a narrower slice?

See it for yourself: FireCompass offers verified CISOPlatform members a free AI-powered pentest of their own attack surface - a direct, no-cost way to answer the five questions above against your own environment instead of a vendor's demo.

How "continuous" is actually defined - and what it isn't

NIST SP 800-115, the technical guide most auditors still measure against, was written around point-in-time assessments: scope a window, run the test, deliver a report, repeat on a schedule. It doesn't mandate a frequency - that's driven by risk, change velocity, and regulatory obligation. PTES fills in the practitioner-level detail NIST leaves abstract. Neither standard was written with continuous, AI-assisted testing in mind, which is exactly why Schneier's framing matters: continuous testing isn't a new standard, it's the same standards-based methodology applied on a rolling basis instead of once a year. If a vendor can't explain how their "continuous" offering still satisfies PTES-style methodology and NIST 800-115 documentation expectations, they're selling a scanner, not a pentest.

Are you actually ready for continuous pentesting? A CISO self-assessment

Do you have a current, accurate asset inventory / attack surface map - or would a continuous scan just continuously find things you didn't know you owned?
Can your remediation process absorb a higher volume of findings without turning into alert fatigue for your team?
Do you have a documented SLA for triaging and fixing findings by severity, one that can keep pace with a rolling test rather than an annual batch?
Is your CI/CD pipeline mapped so new assets are automatically brought into scope, instead of falling into a scanning blind spot?
Do you have a human review step for AI-generated findings before they go to a developer - someone accountable for confirming a finding is real before it becomes a ticket?
Have you told your cyber insurance underwriter you're moving to continuous validation? Several carriers now price risk lower for continuously-tested environments - and ask for evidence of it at renewal.

If you answered "no" to two or more of these, the fix isn't to delay continuous testing - it's to fix the underlying gap first, because continuous testing will expose it either way.

Signals it's time to move now, not next renewal cycle

Your infrastructure changes weekly (new cloud services, CI/CD deploys, M&A-driven environments) faster than your annual test cycle can cover.
A past incident revealed a vulnerability that existed for months between scheduled tests.
You're deploying AI agents or LLM-powered features faster than your security review process can keep up with.
Your compliance framework (PCI DSS 4.0, for instance) already requires more frequent testing than once a year for certain environments.
Your last pentest report read like a template - generic findings, no evidence of manual exploitation, no re-test included.

Questions to ask any continuous / AI pentest vendor (expanded)

Is testing human-led with AI augmentation, or automation-primary with a human signing off at the end? Ask what percentage of hours are manual versus automated.
Can you see the reasoning path behind a finding, not just the finding itself?
Does it automatically re-test previously-fixed issues, or does every cycle start from zero?
What happens when the AI is uncertain about a finding - does it escalate to a human tester or report it anyway?
Will you provide a redacted sample report? Look for CVSS scoring, proof of exploitation, business-impact language, and specific remediation steps - not just a vulnerability list.
What certifications do the testers assigned to your account hold (OSCP, OSCE, CREST, GPEN)? Ask for names and credentials on compliance-driven engagements.
Is one round of retesting included in the price, with a clean attestation letter for your auditors?
How is your data protected at rest and in transit, and when is it purged from their systems after the engagement?
What's the responsible disclosure window if they find something critical - and what's the notification SLA for a critical finding found at 2am on a Saturday?
Which compliance frameworks have they mapped findings to before (SOC 2, PCI DSS, ISO 27001, NIST CSF)?

Red flag worth remembering: a proposal that arrives within 24 hours with no clarifying questions about your environment was built from a template, not from understanding your attack surface.

RFP criteria checklist for continuous / AI-augmented pentesting

Criterion	What to require in the RFP
Methodology alignment	Explicit mapping to PTES and/or NIST SP 800-115, documented in the proposal
Human-to-automation ratio	Disclosed percentage of manual vs. automated testing hours
Tester qualifications	Named testers with certifications (OSCP/OSCE/CREST/GPEN) for compliance-driven scope
Continuous cadence options	Ability to run rolling/continuous testing, not just scheduled point-in-time windows
Re-testing	At least one round of verification retesting included in the base price
Reporting quality	Redacted sample report showing CVSS scoring, proof of exploitation, remediation guidance
Integration	Native or API integration with your ticketing system (Jira, ServiceNow) and SIEM
Data handling	Written data protection, retention, and purge policy for engagement data
Escalation SLA	Defined notification time for critical findings, including after-hours
Compliance mapping	Track record mapping findings to your specific framework (PCI DSS, SOC 2, HIPAA, etc.)
References	At least two reference clients in your industry or of similar environment complexity
Insurance / liability	Proof of E&O / cyber liability insurance covering the engagement

Vendor evaluation scorecard

Score each finalist 1 (poor) to 5 (excellent) on the criteria above, weight the ones that matter most to your environment, and require every vendor to answer the same questions in writing - it's the only way to compare apples to apples instead of comparing sales decks.

One more thing worth knowing

"Pentest theater" - a scan dressed up as a pentest, timed to satisfy an auditor rather than find real risk - is still common enough that it has its own name in the industry. The clearest tell: no manual exploitation, no named testers, and a report that could have been generated for any company in your industry. Continuous testing raises the stakes on this, because a shallow continuous scan run 365 days a year gives you 365 days of false assurance instead of one.

All Articles