All Articles

What PTaaS Actually Means, Technically

Penetration Testing as a Service refers to a delivery model, not a testing technique. The service wraps penetration testing in a platform: a dashboard where findings appear as they are discovered, a subscription or scoped-capacity commercial structure instead of a single invoice per engagement, and a workflow that keeps the client and the testing team in continuous contact rather than exchanging a scope document and a final PDF six weeks later. Traditional consulting-model pentesting is project based. You define a scope, sign a statement of work, wait one to three weeks for a team to be scheduled, and receive a static report at the end of a one to two week testing window, typically once or twice a year.

PTaaS compresses and continuizes that cycle. Instead of a single annual engagement, testing can run on a rolling basis, be triggered on demand before a release, or be tied to a change in the attack surface such as a new subdomain or API endpoint appearing in production. Findings are pushed to a dashboard as they are validated rather than batched into a final report, which means a critical finding discovered on day two of a three-week test is visible and actionable on day two, not three weeks later.

Delivery Cadence: Continuous and On-Demand vs Scheduled Engagement

The practical difference shows up in how testing windows are scheduled. A traditional engagement is booked in advance, staffed for a fixed window, and closed out with a report; running it again means starting the procurement cycle over. PTaaS platforms generally support three cadence models: continuous testing running in the background against a defined scope, on-demand testing that can be requested against a specific application or release with a short lead time, and scheduled recurring testing that runs at a set interval such as monthly or quarterly. The important evaluation question is not whether a vendor uses the word "continuous," but what actually triggers a new test cycle and how fast results land after that trigger fires.

Dashboard-Based Delivery vs PDF-Only Reports

In a PDF-only model, the deliverable is the report itself. There is no interim visibility, no way to track remediation status against individual findings, and no persistent record you can filter or query later. A platform-based PTaaS model treats the report as an export of live data rather than the primary artifact. Findings live in a system that tracks status (open, in remediation, retesting, closed), links to evidence and proof of concept, and can be filtered by severity, asset, or compliance mapping. This matters operationally because security teams need to triage findings the day they appear, not the day the final report is delivered.

Integration With Development Tooling

A defining technical trait of mature PTaaS is that findings do not stay inside the vendor's platform. Findings can be pushed automatically into ticketing systems such as Jira or ServiceNow, with metadata (severity, affected endpoint, CWE ID, proof of concept) attached to the ticket rather than requiring a security analyst to manually re-key it. Some platforms expose a REST API so findings can be pulled into an internal vulnerability management or GRC tool. A smaller but growing subset of platforms support CI/CD triggers, where a pipeline event, such as a deployment to a staging environment, kicks off a targeted automated test pass, with human review layered in for higher-severity signals or before a production release. When evaluating a vendor, ask for the specific list of supported integrations and whether they are bidirectional (status changes in Jira sync back to the platform) or one-way.

Methodology Mix: Where Automation Ends and Human Judgment Begins

This is the single most important technical distinction in the category, and the one most obscured by marketing language. A genuine hybrid PTaaS engagement layers three things: automated reconnaissance and scanning to enumerate the attack surface and flag candidate issues quickly, manual or AI-agent-assisted validation to confirm exploitability and eliminate false positives, and human-led testing for business logic flaws, authorization bypass, and multi-step attack chains that automated tools structurally cannot reason about. Automated scanners are good at finding known vulnerability classes at scale but they cannot understand what an application is supposed to do, so they cannot detect logic flaws such as a discount code that should be single-use but is not, or a workflow that lets a low-privilege user reach an admin function by manipulating a request. A vendor that skips the manual or agentic validation layer entirely and calls the output a "pentest" is running a scanner with a subscription wrapper, not delivering penetration testing.

Building an Evaluation Framework: What to Actually Check

Below is a working checklist for evaluating PTaaS vendors. Treat every claim a sales team makes during this process as a hypothesis to be verified against a sample report, a reference customer call, or a documented methodology, not as a fact.

1. Methodology Transparency

• Ask the vendor to specify, in writing, what percentage of a typical engagement is automated scanning versus human or AI-agent validation versus fully manual testing, and get this broken down by testing type (web app, API, network, cloud, mobile).
• Confirm findings are mapped to a recognized framework: OWASP Top 10, OWASP Web Security Testing Guide (WSTG) test IDs, and CWE identifiers. A mature finding pairs a WSTG test ID with an OWASP Top 10 category and a CVSS score; that structure is what separates a documented finding from a vague claim.
• Ask which testing standard the engagement follows: PTES, OSSTMM, NIST SP 800-115, or an internally documented methodology, and request to see the methodology document itself, not a marketing summary of it.
• Check whether the vendor or its testers hold recognized accreditations. CREST accreditation of the firm itself is a stronger signal than an individual tester holding an OSCP, because it implies documented process, insurance, and organizational accountability rather than one person's skill.

2. False Positive Handling

Ask the vendor directly what their claimed false positive rate is and, more importantly, how that rate is measured. A finding that is "validated" only means the scanner assigned it a severity score based on a CVE or CVSS lookup is not the same as a finding backed by a working proof of concept. Automated scanners running without a validation layer commonly carry false positive rates in the range of 40 to 70 percent because they flag issues based on signature matches or version detection without confirming exploitability in context. A properly validated finding should include reproducible exploitation steps, evidence (screenshots, request and response captures, or a scripted proof of concept), and a description of actual impact, not just a severity label copied from a vulnerability database entry.

3. Retest SLAs

• Ask how many retests are included in the base engagement or subscription, and what happens once that allotment is exhausted (additional cost, delay, or unlimited retesting).
• Ask for the committed turnaround time on a retest request once a fix is marked ready, and whether that SLA differs by severity (for example, critical findings retested within 48 hours versus lower-severity findings retested within a week).
• Confirm retesting is performed by testers with context on the original finding, not a fresh automated scan that might miss the specific exploitation path that was originally used.

4. Integration Depth

• Ticketing: confirm which systems are supported natively (Jira, ServiceNow, Azure DevOps, Linear) versus which require a webhook or custom API work.
• Identity: check whether the platform supports SSO (SAML or OIDC) and role-based access control for who can view findings, since pentest data is sensitive and access needs to be restricted by need to know.
• API access: ask whether findings, assets, and remediation status are available through a documented REST API, and whether that API is rate-limited in a way that would block bulk export into an internal vulnerability management system.
• CI/CD: if relevant to your environment, ask exactly which pipeline events can trigger testing and what scope of testing runs automatically versus what still requires a manual request.

5. Reporting Format and Quality

A report or dashboard export should give you enough detail to reproduce the finding without contacting the vendor. That means: the specific request or interaction that triggers the vulnerability, the affected endpoint or component, a proof of concept (script, curl command, or annotated screenshot sequence), a CVSS score with the vector string shown rather than just the number, the relevant CWE and OWASP Top 10 or WSTG mapping, and remediation guidance specific to the finding rather than a generic paragraph about "implementing input validation." Ask to see a redacted sample report before signing anything. If the vendor cannot produce one, that itself is informative.

6. Data Handling and Compliance

• Ask where test data (scope details, findings, proof-of-concept artifacts, screenshots containing potentially sensitive application data) is stored, and whether that storage location satisfies your own data residency requirements, particularly if you operate under GDPR, or sector-specific rules in finance or healthcare.
• Ask whether the vendor itself holds a SOC 2 Type II report and request to review it under NDA. A vendor asking you to trust them with access to your production or staging environments should be able to demonstrate their own security posture is independently audited.
• Ask how long findings and evidence are retained after an engagement ends, who has access during that retention period, and what the deletion process looks like when the contract ends.
• If testers are contracted rather than direct employees, ask how background checks and confidentiality agreements are handled for third-party testers who will have access to your systems.

7. Scalability and Pricing Structure

PTaaS pricing generally follows one of a few structures: per-application or per-asset pricing, per-scope pricing that bundles a defined set of assets into a fixed engagement price, or subscription pricing that grants a pool of testing capacity (measured in hours, test credits, or number of assets) over a period. Ask how the price changes as your application or API inventory grows, since a pricing model that works cleanly for ten applications can become unpredictable at fifty if pricing is not clearly tiered. Ask specifically how new assets are onboarded mid-contract: is there a defined process for adding an application to scope without renegotiating the entire agreement, and how quickly can testing start on a newly added asset.

Comparison: Three Delivery Models

The table below lays out the three broad categories a buyer will encounter. Few vendors sit at a pure extreme; use this as a reference for where a given vendor's actual delivery falls, based on verified answers to the checklist above rather than marketing copy.

A CISO Platform member perk. FireCompass is offering CISO Platform members a free AI pen test on their own attack surface. Run yours and judge the delivery model for yourself.

Red Flags: Signs a Vendor Is Rebadging a Scanner as PTaaS

The label "PTaaS" carries no enforced technical definition, so the category includes vendors running little more than a commercial vulnerability scanner behind a dashboard and a subscription price. Watch for the following signals during evaluation.

1. No manual or agentic validation step is described anywhere in the methodology documentation, and the sales team cannot explain, in specific terms, what a human or an AI agent actually does with a raw scanner output before it reaches you as a "finding."
2. Findings consist of a CVE identifier, a CVSS score pulled from a public database, and a generic description, with no proof of concept, no request or response capture, and no explanation of how the vulnerability was confirmed to be exploitable in your specific environment.
3. Retesting is not included, is billed separately at a meaningful cost, or is itself just an automated rescan that checks for the same signature rather than confirming the original exploitation path is closed.
4. There is no available methodology document, testing standard reference (PTES, WSTG, NIST 800-115), or sample report, and requests for one are deflected with marketing language instead of specifics.
5. The sales team, when asked directly what percentage of the engagement is automated versus human-reviewed, cannot give a straight answer or gives an answer that shifts between conversations.
6. The vendor cannot name the testers or tester qualifications assigned to your engagement, or refuses to share accreditation details (CREST, individual certifications) for the team doing the work.
7. Business logic testing, authorization and access control testing, and multi-step attack chaining are absent from the scope description, with the engagement limited to what an off-the-shelf scanner can detect on its own.
8. Pricing scales in a way that is opaque as your asset count grows, with no clear published or contractual structure for onboarding new applications or APIs mid-term.

Questions to Ask a PTaaS Vendor During Evaluation

Use these as direct questions in a vendor call or RFP, not rhetorical ones. A vendor confident in their delivery model will answer specifically; a vendor rebadging a scanner will answer vaguely or redirect to a case study.

1. What percentage of findings in a typical engagement come from automated scanning alone versus manual or AI-agent validated testing, broken down by test type?
2. Can you show me a redacted sample report or dashboard export from an engagement similar in scope to mine, including at least one finding with full proof of concept?
3. What is your measured or claimed false positive rate, and what specific process produces that number?
4. How many retests are included, what is the SLA for turnaround on a retest request, and what happens once the included retests are used?
5. Which ticketing systems, SSO providers, and CI/CD platforms do you integrate with natively, and which require custom API work on our side?
6. Is your organization CREST accredited or equivalently certified, and what certifications do the specific testers assigned to my account hold?
7. Where is our test data and evidence stored, for how long after the contract ends, and can you provide your own SOC 2 Type II report under NDA?
8. How does your methodology handle business logic testing and multi-step attack chains that a scanner cannot detect on its own?
9. How does pricing change as we add applications or APIs to scope over the course of the contract, and is there a defined onboarding process for new assets?
10. Walk me through exactly what happens between the moment your platform detects a candidate vulnerability and the moment it appears as a finding in my dashboard.

Putting It Together

PTaaS is a legitimate evolution in how penetration testing is delivered, and for organizations shipping code frequently or managing a growing number of applications and APIs, the cadence and integration advantages over a purely annual, PDF-based engagement are real. The risk is not the delivery model itself but the fact that the label has been adopted broadly enough that it no longer guarantees any particular level of manual rigor. The checklist, comparison table, and red flags above are meant to be used mechanically during a vendor evaluation: ask the specific questions, request the specific artifacts, and weight the answers over the pitch. A small number of vendors in this space, including names like Cobalt, HackerOne, BreachLock, Synack, and FireCompass among others, have published enough methodology detail publicly to be evaluated against this framework directly, and doing that comparison yourself will tell you more than any single vendor's marketing page.

As a CISO Platform member, you get free access. Ready to see agentic PTaaS on your own attack surface? Start your free AI pen test.

About Priyanka Aash
Priyanka Aash is Co-Founder of CISO Platform, the world's first online community for information security executives, and Co-Founder of FireCompass. She has been nominated for the Cybersecurity Excellence Award for leadership and AI innovation in cybersecurity, honored with the NetApp Excellerate HER award, and featured in SC Media's Women in IT Security series. She is the author of The AI Divide. Security technologist Bruce Schneier advises FireCompass.