All Articles

The math of an annual test cycle

Start with the arithmetic a CISO actually has to defend in a board meeting. If an environment is tested once a year, and the test itself takes one to three weeks to scope, execute, and report, the organization has current, validated knowledge about its security posture for roughly 2-6% of the calendar year. For the remaining 340-plus days, every finding in that report is aging, and every day after the test is a day of drift the report never accounted for.

That drift is not hypothetical. A typical engineering organization running weekly or biweekly release cycles ships dozens to hundreds of code changes between one annual test and the next. New services get spun up in staging and forgotten there. Dependency updates pull in transitive packages nobody explicitly vetted. Cloud infrastructure changes through Terraform applies and console clicks that never touch a change-management ticket. DNS records get added for a marketing campaign and never removed. Each of these is a potential new entry point, and none of them existed when the annual test was scoped.

The result is a coverage math problem, not just a frequency problem. A point-in-time test can only certify the state of the system at the moment it ran. It says nothing about the API endpoint shipped four months later, the subdomain that appeared when a new SaaS integration went live, or the S3 bucket a contractor left with public read access after a proof-of-concept. The report is accurate on the day it is delivered and progressively less representative of reality every day after that.

What the current threat data says about the exposure window

The gap between annual testing and actual attacker speed is no longer a matter of opinion; it shows up directly in the two most-cited breach data sets in the industry.

According to Mandiant's M-Trends 2026 report (released March 2026, based on more than 500,000 hours of frontline incident response in 2025), average time-to-exploit for disclosed vulnerabilities has gone negative. The report tracks this trend over time: average time-to-exploit was 63 days in 2018, fell to -1 day by 2024, and reached an estimated -7 days in 2025. In practical terms, exploitation is now routinely observed before a patch is even public, meaning threat actors are reverse-engineering fixes, exploiting pre-disclosure leaks, or independently discovering the same flaw researchers are about to publish. Exploits remained the single most common initial infection vector for the sixth consecutive year running, accounting for 32% of intrusions Mandiant investigated in 2025.

The same report found that global median dwell time, the time between initial compromise and detection, rose to 14 days in 2025 from 11 days in 2024. For cyber espionage intrusions and North Korean IT worker cases specifically, median dwell time was 122 days. Perhaps more telling for anyone thinking about response speed: the median time between an opportunistic initial-access event and hand-off to a second, more damaging threat group collapsed from more than 8 hours in 2022 to just 22 seconds in 2025, because initial-access brokers now pre-stage the follow-on group's tooling before the hand-off even happens.

Verizon's 2026 Data Breach Investigations Report (the 19th edition, published May 2026, based on more than 31,000 incidents and 22,000 confirmed breaches across 145 countries) independently confirms the same shift from the breach side. Vulnerability exploitation now accounts for 31% of breaches, surpassing stolen credentials as the top initial access vector for the first time in the report's 19-year history, up from 20% the year before. Credential-related vectors, phishing, credential abuse, and pretexting combined, still account for roughly 32%, so the two categories are now essentially tied at the top, but exploitation's growth rate is the more urgent trend for defenders to plan around.

The same DBIR documents the other half of the problem: remediation is not keeping pace. Median time to fully patch a vulnerability stretched to 43 days in the 2026 report, up from 32 days the year prior, and only 26% of critical flaws listed in CISA's Known Exploited Vulnerabilities catalog were fully remediated by the organizations studied, down from 38% the year before. Attackers are weaponizing flaws before or within days of disclosure while organizations take over six weeks, on average, to close them. An annual pentest, run once against a snapshot of that patch backlog, is testing a moving target with a camera that takes one photo a year.

What continuous testing catches that an annual cycle structurally cannot

The value of frequent testing is easiest to see in concrete scenarios rather than abstractions. These are the kinds of gaps that show up in real engagements between annual test cycles.

Scenario 1: the mid-quarter API with a BOLA flaw

A product team ships a new mobile app feature in month four of the annual test cycle. It exposes a new REST endpoint, /api/v2/orders/{order_id}, that pulls order details by ID. The endpoint checks that the requesting user is authenticated, but not that the authenticated user actually owns the order being requested, a textbook broken object level authorization (BOLA) flaw. Any logged-in user can enumerate order IDs and read other customers' order history, shipping addresses, and partial payment data. This endpoint did not exist when the annual test was scoped in month one. It will not be tested until month thirteen, when the next annual cycle begins, unless something in between catches it.

Scenario 2: the dependency CVE that goes from disclosed to exploited in days

A widely used logging or serialization library discloses a remote code execution vulnerability. Within 48-72 hours, proof-of-concept exploit code is public, and mass scanning for vulnerable instances begins within the week, a pattern consistent with the accelerating time-to-exploit trend Mandiant has tracked since 2018. The organization's annual pentest happened two months earlier and, naturally, tested the application with the vulnerable version of the library already in place, because the CVE hadn't been disclosed yet. Nobody re-tests the application against the new disclosure until the next scheduled cycle, ten months away, even though the exploit is public and the library is sitting in production right now.

Scenario 3: the storage bucket misconfigured three months after the test

A data engineering team spins up a new cloud storage bucket for a reporting pipeline. During initial setup, someone sets the bucket policy to allow public read access "temporarily" to unblock a demo, then moves on without reverting it. The annual pentest, run three months prior, never saw this bucket because it did not exist yet. Attack surface management tooling that continuously enumerates cloud assets and checks configuration state would flag the public bucket within hours of the policy change. A calendar-driven annual test would not touch it until the next scheduled engagement, whenever that falls, and only if anyone remembers to include that asset in next year's scope at all.

Scenario 4: the "fixed" vulnerability that regresses

A finding from an earlier engagement, say, a stored cross-site scripting flaw in a comment field, gets remediated and closed. Six months later, a refactor of the comment rendering component reintroduces the same class of flaw because the new developer on the team was not aware the original fix existed or why it mattered. Without a regression check tied to that specific finding, the vulnerability is effectively rediscovered from zero the next time someone happens to test that code path, which might be the next annual cycle, or might be an attacker.

In each case, the failure is not that annual testing is bad at what it does. It is that the model tests a fixed scope on a fixed calendar, while the actual attack surface changes continuously and asynchronously to that calendar.

A CISO Platform member perk. FireCompass is offering CISO Platform members a free AI pen test on their own attack surface. Run yours and see how fast findings surface against your real environment.

How continuous testing operates in practice

"Continuous" is frequently used loosely to mean "we scan more often." Operationally, a testing program that deserves the label is built around triggers, scoping discipline, and a defined finding lifecycle, not just a shorter interval between full-scope engagements.

Trigger models

Mature continuous testing programs run on a combination of three trigger types rather than one fixed schedule:

• Deploy-triggered testing: A CI/CD pipeline event, typically a deploy to staging or production, fires a scoped test against the routes, endpoints, or services that changed in that release, rather than re-running a full-surface test on every commit. This is what catches the mid-quarter API endpoint from scenario 1 close to when it ships, instead of ten months later.
• Disclosure-triggered testing: A new CVE affecting a library, framework, or component in the environment triggers a targeted retest of every asset known to use that dependency. This requires the testing program to maintain a live software bill of materials or dependency inventory; without it, there is no way to know which of the organization's hundred-plus applications need an emergency check when a new CVE drops.
• Drift-triggered testing: Attack surface management tooling continuously enumerates external assets, subdomains, open ports, cloud storage configurations, and certificate changes, and triggers a test whenever it detects something that was not present in the last known-good inventory. This is what catches scenario 3's exposed bucket and the shadow IT that a code-based trigger would never see, because it was never part of a deploy in the first place.

A defined re-test cadence, often tiered by application criticality, remains the fallback for assets that do not generate their own triggers: a payment API might get deploy-triggered testing on every release, while a low-traffic internal tool defaults to a monthly or quarterly sweep. The tiering matters because testing everything at maximum frequency is neither affordable nor useful; the goal is matching frequency to risk, not maximizing frequency uniformly.

Delta-based scoping

The technical prerequisite that makes deploy- and drift-triggered testing viable at all is a live, versioned asset inventory tied to the testing engine, so the system can diff the current state against the last tested state and scope the next test to what actually changed. Without that inventory, every trigger degenerates into a full-surface retest, which is not sustainable at deploy frequency and just becomes "frequent" rather than "continuous." Delta-based scoping is also what keeps a continuous program from producing an unmanageable volume of low-value findings: re-validating unchanged code paths on every trigger produces noise, not signal.

Regression testing of prior findings

Every finding that gets marked "remediated" needs a scheduled or trigger-based retest against that specific finding, not just a general assumption that the fix held. Scenario 4's regressed XSS flaw is exactly the failure mode this catches: the fix was correct at the time, but nothing verified it stayed correct after a later refactor touched the same code. Regression checks are typically cheap to run relative to a full test, since they are scoped to one specific finding and one specific code path, which makes them a good candidate for high-frequency automated execution even in programs that reserve deeper manual testing for lower frequency.

Finding lifecycle

A continuous program lives or dies on whether findings move through a disciplined lifecycle instead of accumulating in a spreadsheet. The stages that matter:

1. Discovered: A trigger-based or scheduled test identifies a candidate finding.
2. Validated with proof-of-concept: The finding is confirmed exploitable, not just flagged by a signature or heuristic. This step is what separates continuous testing from continuous scanning; a scanner that fires on a tight loop without validation just produces more unverified alerts, not more actionable signal.
3. Triaged and assigned: The validated finding is scored for business impact, given an owner, and given a remediation SLA tied to severity.
4. Remediated: Engineering ships a fix.
5. Retested: The specific finding is retested against the fix, not assumed closed because a ticket was marked done.
6. Closed: The finding is closed only after retest confirms the fix holds, and it re-enters the regression pool for future spot checks.

Skipping the validation step is the most common failure mode when organizations try to bolt "continuous" onto existing vulnerability scanning without changing the underlying process. Skipping the retest step is the most common failure mode when organizations have a testing program but no regression discipline. Both failures are process gaps, not tooling gaps, which is why governance matters as much as coverage.

Annual, quarterly, and continuous compared

Objections security teams raise, and grounded responses

"We don't have budget for this."

The budget objection usually assumes continuous testing means running the same manual, per-engagement pricing model twelve times instead of once, which would indeed be unaffordable for most portfolios. That is not how continuous programs are typically priced or built. Automated discovery combined with validated testing brings per-application cost down substantially compared to fully manual engagements, which is what makes testing a portfolio of 200 to 2,000-plus applications a realistic annual number rather than an aspirational one. The more useful framing for a CISO is not "what does continuous cost versus annual" but "what is the cost of the exposure window annual testing leaves open," measured against the DBIR's finding that vulnerability exploitation is now the leading breach vector.

"Continuous testing means continuous alert fatigue."

This objection is valid against continuous scanning, not continuous testing, and the distinction is the whole point. A signature-based scanner run on a tight loop against an unchanged environment produces the same false positives over and over, just more frequently. A continuous testing program that validates findings with proof-of-concept before they reach a security team's queue does not have this problem by construction, because unvalidated noise never reaches the queue in the first place. If a continuous program is generating fatigue, the fix is adding a validation gate, not reducing test frequency.

"We already do vulnerability scanning, isn't that the same thing?"

Vulnerability scanning and penetration testing answer different questions. A scanner checks whether a known signature, missing patch, or misconfiguration pattern is present; it does not chain findings together, does not confirm exploitability in context, and does not test business logic flaws like the BOLA scenario described earlier, where every individual control (authentication) works correctly and the flaw only exists in how those controls compose. Scanning is necessary and cheap to run frequently; it is not a substitute for testing that validates whether a chain of findings actually produces exploitable access. A mature continuous program runs both: frequent scanning for known-signature issues, and continuous, validated testing for exploitability and logic flaws scanning cannot see.

"Our compliance requirement only asks for an annual test."

Compliance frameworks that specify annual testing are typically defining a floor, not a ceiling, and satisfying that floor does not mean the exposure window described earlier goes away between audits. An organization can maintain a continuous program for its own risk management purposes while still producing the point-in-time annual report a specific framework requires; the two are not mutually exclusive; continuous testing over the course of a year makes the annual compliance report a more accurate reflection of a well-managed environment rather than a lucky snapshot.

As a CISO Platform member, you get free access. Ready to see what continuous testing surfaces on your attack surface? Start your free AI pen test.

About Priyanka Aash
Priyanka Aash is Co-Founder of CISO Platform, the world's first online community for information security executives, and Co-Founder of FireCompass. She has been nominated for the Cybersecurity Excellence Award for leadership and AI innovation in cybersecurity, honored with the NetApp Excellerate HER award, and featured in SC Media's Women in IT Security series. She is the author of The AI Divide. Security technologist Bruce Schneier advises FireCompass.