Continuous pentesting, BAS, autonomous pentesting and CART: what actually validates external exposure

Four product categories now compete to answer the same question, and they describe themselves in almost the same words. Breach and attack simulation, autonomous pentesting, continuous penetration testing and continuous automated red-teaming each promise proof rather than a list — evidence that a weakness is real, reachable and usable, rather than a scanner's opinion that it might be.

They are not substitutes for one another. Each begins from a different starting position, tests a different thing, and is blind in a different direction. The expensive mistake is rarely buying a bad product; it is buying a good one that answers a question you did not have. A team that acquires control-efficacy testing when what it needed was external discovery ends up with clean dashboards and an incident on an asset that was never in scope.

What follows compares categories rather than named products. If you are further along and building the evaluation itself, the platform evaluation hub covers scoring, scope and the questions worth asking.

Validation is four different questions

Analysts have begun grouping all four under a single heading — adversarial exposure validation — which is useful for budgeting and misleading for buying. The word "validation" has been stretched to cover four distinct enquiries, and most disagreements about these tools are really disagreements about which question was being asked:

1. Do my controls detect and stop known attacker techniques?
2. If an attacker had a foothold, how far could they get?
3. Can a skilled human find a way in, repeatedly, on a schedule?
4. Of everything visible from outside my perimeter, what is actually usable today?

Each category below answers one of those well and the others poorly.

Breach and attack simulation: control efficacy

Breach and attack simulation executes a library of known adversary techniques — typically mapped to MITRE ATT&CK — inside your environment in a controlled way, and measures what your controls did about it. Was the technique blocked? Detected? Logged and forwarded to the SIEM? Did the alert actually fire, and did it reach anyone?

This is a genuinely valuable instrument, and it is the only one of the four that measures your defensive stack rather than your attack surface. Teams that run it seriously find configuration drift they had no other way to see: an EDR policy that silently reverted, a log source that stopped shipping in March, a detection rule that matched on a field the vendor renamed.

Its boundary is definitional. Breach and attack simulation tests the estate you point it at, using the techniques its library contains. It is not a discovery mechanism and does not claim to be. Coverage is therefore bounded twice — by the technique library, and by your asset inventory.

That second bound matters more than most buyers expect. Across ShadowMap deployments we typically surface 30 to 60 per cent more external assets than the customer's own inventory contained. A control-efficacy programme running against the inventory you already had is, by construction, silent about all of them. The dashboard is not wrong; it is answering a narrower question than the one you thought you asked.

Autonomous pentesting: attack paths from a foothold

Autonomous pentesting automates the reasoning a network penetration tester performs: start from a position, chain misconfigurations, credential reuse and known vulnerabilities together, and reach an objective — domain administrator, a sensitive data store, a privileged service account.

At this it is very good, and better than a human at one specific thing: repeating the exercise weekly without fatigue. Path-finding through Active Directory, over-permissioned service accounts and reused local administrator passwords is combinatorial work, which is exactly the kind of work automation should be doing.

The important qualifier is where the starting position sits. Most of the demonstrable value in this category is inside the perimeter, because that is where the chains are. An automated red teaming engine pointed only at a hardened external estate has comparatively little to chain — a handful of exposed services, most of them patched, and no lateral movement to reason about. The category is frequently sold as external validation and delivers most of its value internally.

One question is worth putting directly to anyone selling it: what does the engine actually execute against production, and what does it infer? Genuine exploitation carries genuine risk, and vendors resolve that tension in different and legitimate ways. The resolution should be explicit in the contract rather than discovered during an outage.

Continuous penetration testing: humans, on a cadence

Continuous penetration testing replaces the annual engagement with recurring human-led testing, usually delivered through a platform that manages scoping, findings, retest and evidence. The testing itself is still performed by people; the "continuous" refers to cadence and workflow rather than to automation.

This is the only category of the four that finds business-logic flaws. No technique library contains "the password-reset flow accepts a user ID from the request body", or "the tenant identifier in this API call is trusted from the client", or a chained authorisation failure that requires understanding what the application is for. Those findings come from a human who read the application, and they are frequently the most serious issues in an external estate.

Its limit is scope, and it is structural rather than a matter of quality. A penetration test assesses what you asked it to assess; discovery is not the product. If the target list is incomplete, the test is rigorous against the wrong estate. The forgotten subsidiary domain, the marketing microsite a contractor registered, the staging environment a certificate transparency log has been publishing for two years: none of these are findings in a scoped engagement, because none were in the scope document.

Cadence is the second constraint. Human testing is expensive, so it is bounded, so it happens in windows. External exposure does not respect windows. A repository goes public on a Tuesday; a credential appears in a stealer log the following week.

Continuous Automated Red-Teaming, as we build it

The term is used loosely across the market, so here is precisely what it means in ShadowMap rather than what it means in general.

Validation is applied to artefacts the platform itself discovered, from outside, without prior knowledge, credentials or agents. The sequence matters: discover the estate, attribute it to you, surface the exposures — assets, leaked source code, exposed keys, credentials and stealer-log material, brand abuse, dark-web mentions, vendor posture — and then validate the classes where validation is safe, authorised and decisive.

Concretely, that means establishing whether an exposed key is live and what it opens; whether a leaked credential still authenticates against an in-scope authentication surface, and what access it carries when it does; whether an exposed administrative interface is genuinely reachable; whether a misconfiguration actually yields data or merely looks as though it might.

The differentiator is not the act of validation, which several categories perform. It is that validation runs inside a platform that already holds the discovered assets, the leaked code, the exposed keys, the credential and stealer-log corpus, the brand-abuse signals and the vendor context — so a validated finding arrives already correlated to everything else known about that identity, that host and that supplier. A validated credential is not a row; it is a credential that reached a named service, on a host discovered three weeks earlier, belonging to a device that also leaked live session cookies.

Two numbers describe the shape of the output. Across deployments, between 8 and 15 per cent of surfaced exposures validate as genuinely exploitable. The useful reading of that range is its remainder — the other 85 to 92 per cent is real exposure that does not need to be worked this week, and knowing which is which is what moves time-to-action on high-severity findings by 40 to 60 per cent. In the first thirty days we typically surface between 5 and 15 active secrets — keys and tokens that are live, not expired credentials in an old commit.

The comparison

	Starting point	What it proves	What it requires	Where it is blind
Breach and attack simulation	Assets you nominate, inside your environment	Your controls detect, block and log known techniques	Deployment inside the estate; a maintained technique library	Assets not in your inventory; anything outside the perimeter
Autonomous pentesting	A foothold, network range or credential set	An attacker with that position could reach a defined objective	Internal access; agreed exploitation boundaries	External-only estates; exposure that is not a network path
Continuous penetration testing	A scope document you supply	A skilled human could or could not get in, including business-logic routes	Human time; a defined and current scope	Anything outside scope; anything that changes between windows
Continuous Automated Red-Teaming (external)	Nothing — discovery is the first step	Which externally discovered exposures are actually usable	Contractual authorisation and progressive scoping	Internal estate; anything requiring agents or endpoint telemetry

What none of these replaces

A scoped human red team. Automated validation confirms classes of exposure it has been built to confirm. An objective-driven red team tests the organisation — people, process, detection, escalation, decision-making under pressure — against a goal, using whatever route presents itself, including ones nobody anticipated. That is a different exercise, and no amount of automation produces it.

We are unusually placed to say so plainly. ShadowMap is built by Security Brigade, which has delivered more than 6,700 security assessments since 2006 and has been CERT-In empanelled since 2008. We sell human testing and we sell automated validation, which means we have no commercial reason to pretend either one is the other. When the right answer to a question is a scoped human engagement, that is the answer we give.

Internal vulnerability management. None of the external categories tells you the patch level of a server inside your network, and none should claim to.

The hard boundary: what our validation does not do

Claim discipline is part of the product, so the limits are stated rather than buried.

Validation is not enabled at go-live. Onboarding begins with discovery and enumeration; your inventory is built and confirmed first, and validation is introduced progressively within the scope established by the agreement and the onboarding configuration. Authorisation is contractual, not assumed.

We do not attempt authentication against systems outside your authorised scope. We do not run public exploit code against your production estate. We do not test credentials belonging to your customers — those remain exposure worth knowing about, reported honestly as untested rather than dressed up as validated.

We do not validate every category, and neither does anyone else. Phishing infrastructure, brand mentions, threat-actor reporting and media monitoring cannot all be validated in the sense the word implies. Any vendor claiming every signal in every category arrived pre-validated is describing something that cannot be done.

We also publish no accuracy percentage for AI Review, deliberately. A single figure spanning every capability and wildly different finding types would be a marketing number rather than a measurement. If a vendor quotes you one, ask what the denominator was and who chose it.

We are external. No agents, no endpoint telemetry, no Active Directory path-finding. If your unanswered question is internal lateral movement, this is not the category you need.

Choosing between them

If your controls are the uncertainty — you have the inventory, you have the stack, and you do not know whether it works — breach and attack simulation answers that and the others do not. If your concern is what an attacker does after landing inside, autonomous pentesting is the discipline built for it. If you need business-logic assurance on commercially significant applications, that remains human work on a cadence.

If you cannot state with confidence what you own on the internet, all three are scoped against an inventory you have not verified — and that is the problem external validation solves first. Most mature programmes run two of these rather than one. The failure worth avoiding is buying the second in the belief that it covers the first.

When you get to the evaluation itself, test discovery before you test validation, because validation quality is meaningless on an estate the platform never found. Our guide to scoping a fourteen-day proof of concept sets out how to measure that in practice, and the alternative-stack cost comparison covers what the same coverage costs assembled from separate tools. When you are ready to specify what you need, start here.

See what a validated finding actually looks like. Sample CART output — the evidence attached to a confirmed exposure, the state model behind it, and the scope boundaries it was produced within. → See a CART validation sample

Related: Continuous Automated Red-Teaming · How to scope a 14-day exposure POC · Evaluate an external exposure platform