AI pentest is two jobs pointing in opposite directions
One reading puts the AI in the tester’s chair: a platform maps the application, works out the test cases that application needs, tries them and proves what it finds. The other puts the AI in the target: a model inside somebody’s product, the things that product retrieves and trusts, and the tools it can be made to invoke. B-52 does both. This article exists so that you get the one you came for.
Yash K · · Autonomy and its limits · about 13 min
The ambiguity
Four words, two sentences, opposite meanings
The phrase does not say which side of the test the AI is on. The two readings need different scoping, different access and different people.
Read it one way and the AI is the tester. A platform maps an application, works out the test cases that application needs, tries them, proves what it can and writes the result up, with a person authorising the scope and — in the fully autonomous model — doing nothing else. Read it the other way and the AI is the target. A model sits inside somebody’s product; the application around it retrieves things and treats them as trustworthy, renders output into a page, or hands a tool call to something holding permissions of its own, and the job is to find where that goes wrong. Both are real engagements, both get bought under the same four words, and neither is a subset of the other. The tester-side reading is a statement about method: how the work is done, and by what. The target-side reading is a statement about scope: what is in front of the tester when the work starts. A request that does not say which one it means cannot be scoped, and the cost of guessing is a confident report answering a question nobody asked.
Which one you arrived with
Three rows, and the third one is both
The third row is the one people do not expect: a product built on a model, assessed by a platform that is itself model-driven. It is an ordinary scope rather than an edge case.
| State | What it means | What follows |
|---|---|---|
| The tester is the AI | The platform maps your application, derives the checks that application needs rather than running a fixed list, attempts them where it is authorised to, and writes up what it proved. Your application is the subject; the autonomy is the method, and it is not part of the scope. | B-52 itself. The platform pages carry it. |
| The target is the AI | Prompt injection — direct, and indirect through whatever the application retrieves and trusts. The interface in front of the model: what it accepts, what leaks through it, what it costs to abuse. And agentic behaviour, where the tool call is the attack surface and privilege escalation runs through what a tool is permitted to do. | Coverage class eleven, LLM applications. |
| Both, in one engagement Terminal | Your product is an LLM application and you want it assessed without booking a team. The platform performs the test and the model-backed application is the target, in the same run, under one scope sign-off. | One scope, one run, one report. |
- Autonomy as the method — the platform doing the testing
- Autonomy as the target — a coverage class where the AI is what is assessed
- Both at once, which is a normal scope and not a special case
- TerminalNo state follows this one
Side by side
Six questions that separate the two readings
Answer the first row and the other five follow from it. That is the whole reason the first row is worth a sentence in the request.
| The question | When the AI is the tester | When the AI is the target |
|---|---|---|
| What is being assessed | Your application, API, network, cloud estate, Active Directory or source code. The autonomy is how the work gets done and is not itself in scope. | The application built around a model — what it retrieves and treats as trustworthy, what it renders, and which tools it can be persuaded to invoke. |
| Every other row on this table is a consequence of this one. It is the sentence missing from a request that cannot be scoped. | ||
| What you supply | Scope sign-off, and a credential per role wherever the target is authenticated, so that the authorisation decisions behind each role can be worked rather than guessed at. | The same, plus the interface in front of the model and whatever the application is permitted to retrieve and act on. |
| Which coverage class | Any of the eleven. Web application, mobile app, API, thick client, external network, internal network, cloud, Active Directory, social engineering, secure code review, LLM applications — the method does not change with the class. | LLM applications, which is the eleventh, and the API class beside it wherever the interface in front of the model is an API, because everything that class works applies to it. |
| What arrives with a finding | The request as sent and the response as returned, the steps that reproduce it, a CVSS v4.0 vector and a CWE. | The same artefact, and the bar is the reason it is the same. A model being talked into an unwelcome sentence is not a finding until something acts on it. |
| Whose signature the report carries | Fully autonomous carries none. Autonomous expert-verified and human-led both carry the verification of a senior Security Brigade auditor. | Identical. All three delivery models cover all eleven classes; what differs between them is whose signature the report goes out under. |
| Where it lives on this site | The page written for the tester-side reading | LLM penetration testing, the eleventh coverage class |
| A third page sits beside both. Agentic penetration testing carries the positioning word rather than either reading. | ||
What is being assessed
- When the AI is the tester
- Your application, API, network, cloud estate, Active Directory or source code. The autonomy is how the work gets done and is not itself in scope.
- When the AI is the target
- The application built around a model — what it retrieves and treats as trustworthy, what it renders, and which tools it can be persuaded to invoke.
Every other row on this table is a consequence of this one. It is the sentence missing from a request that cannot be scoped.
What you supply
- When the AI is the tester
- Scope sign-off, and a credential per role wherever the target is authenticated, so that the authorisation decisions behind each role can be worked rather than guessed at.
- When the AI is the target
- The same, plus the interface in front of the model and whatever the application is permitted to retrieve and act on.
Which coverage class
- When the AI is the tester
- Any of the eleven. Web application, mobile app, API, thick client, external network, internal network, cloud, Active Directory, social engineering, secure code review, LLM applications — the method does not change with the class.
- When the AI is the target
- LLM applications, which is the eleventh, and the API class beside it wherever the interface in front of the model is an API, because everything that class works applies to it.
What arrives with a finding
- When the AI is the tester
- The request as sent and the response as returned, the steps that reproduce it, a CVSS v4.0 vector and a CWE.
- When the AI is the target
- The same artefact, and the bar is the reason it is the same. A model being talked into an unwelcome sentence is not a finding until something acts on it.
Whose signature the report carries
- When the AI is the tester
- Fully autonomous carries none. Autonomous expert-verified and human-led both carry the verification of a senior Security Brigade auditor.
- When the AI is the target
- Identical. All three delivery models cover all eleven classes; what differs between them is whose signature the report goes out under.
Where it lives on this site
- When the AI is the tester
- The page written for the tester-side reading
- When the AI is the target
- LLM penetration testing, the eleventh coverage class
A third page sits beside both. Agentic penetration testing carries the positioning word rather than either reading.
Requests, as they arrive
Seven sentences that sound alike and route differently
Under each one is the reading it points at, and the page that answers it. One of them points at neither, and that is the useful thing about it.
“We need a pentest of the assistant we shipped.”
Coverage class eleven. The work is the application around the model: what it retrieves and trusts, what it renders into a page or a query, and which tools it can be made to invoke on somebody else’s behalf.
The tester is the AI“Can your AI run our quarterly test?”
The fully autonomous model. You sign off the scope and nothing further is asked of you; the observed median turnaround on that model is one to three business days, which is a median taken from real runs rather than a service level.
“We want AI red teaming.”
This one splits, and the split is not predictable from the words. It can mean an adversarial assessment whose target is a model, and it can mean that the testing itself should be automated. It is the phrase worth asking back about before anything is scoped.
“Our auditor wants the AI feature assessed.”
The word “auditor” settles two things at once. The target is the model-backed feature, and a report that has to carry a signature needs the expert-verified or human-led model, where a senior Security Brigade auditor verifies every finding before it reaches you.
The tester is the AI“How far is the automation allowed to go?”
The right question to ask about the tester-side reading, and the one worth asking first: what it may attempt, where it stops, and what it will not do without being told.
Both at once“Our product is an LLM application and we want it tested without booking a team.”
Both readings in one engagement. The platform performs the test and the model-backed application is the target, and that combination is scoped the way any other coverage class is scoped.
A method question“What is the difference between agentic and autonomous testing?”
A question about method rather than about target, which puts it on the tester side of the line before it has been answered. The page carrying the positioning word is the shortest route to it.
Naming
Why the page about testing AI carries an LLM-prefixed name
It is a deliberate choice and it is worth explaining, because the naming is also the clearest short proof that the two meanings are held apart here.
When two of our own pages could plausibly take one phrase, one of them has to give it up, because the alternative is that a reader gets whichever of the two a search engine guessed. The tester-side reading took the AI-prefixed page, since that is what B-52 is and that is where somebody arriving on the method question needs to land. So the coverage class where the target is the AI is named for large language model applications rather than for AI, because a second AI-prefixed page on this site would be two of our own pages competing over one ambiguous term. The LLM-prefixed name has a second virtue, which is grammatical. In “LLM penetration testing” the object of the sentence is unmistakably the thing being tested, and there is no second reading to fall into. A third page beside both carries the positioning word, agentic penetration testing, and describes what the platform is rather than which side of the test the AI sits on. This article takes no term of its own at all. It exists to send you to the right one of the three, and that is the entire job.
One request, three desks
The same four words, read by three people who need different things from them
The ambiguity is cheap to resolve at the first desk, awkward at the second and expensive at the third.
The person writing the request
- The words
- “We need an AI pentest before the release.”
- The job underneath
- Either one. There is a deadline and a budget line, and the phrase was borrowed from whoever asked for it upstream — a board paper, a customer questionnaire, a contract clause.
- What settles it
- One added sentence: is the AI doing the testing, or being tested? Nothing else about the request has to change, and it is the only point at which the correction is free.
The person scoping it
- The words
- The same sentence, now attached to a target list and a date.
- The job underneath
- Two different scopes. The tester-side reading needs a target and a credential per role. The target-side reading needs the interface in front of the model, and whatever the application is permitted to retrieve and act on.
- What settles it
- Naming the coverage class out loud. Eleven exist and only one of them has a model as its target, so the class name carries the answer that the four words did not.
The person the report goes to
- The words
- A finished report, and a question about whether it covers what was asked for.
- The job underneath
- A report on the wrong reading is not a bad report. It is a sound report about something else, which is harder to notice and later to discover than an obviously weak one.
- What settles it
- Nothing, by this point. The scope was signed, the run is done, and the remedy is a second engagement. Which is why the first desk is where this belongs.
Sourcing
Where the 90–95% figure comes from, and what it is not scoped to
It belongs to the tester-side reading. It is a proportion of a findings set rather than a score, and it is stated here with its denominator because that is the only form in which it means anything.
The 90–95% figure As of 2026-09-15
- B-52 and Security Brigade’s expert assessment team worked the same targets at the same time, and neither side saw the other’s output while the work was running.
- Both sets of findings were pooled into one denominator with each item counted once, so a defect both sides reached counts once rather than twice.
- B-52 reached 90–95% of that pooled set. Part of what it reached was absent from the team’s own output, which is why the pooled set is larger than either side produced alone.
- Every engagement Security Brigade has run since the firm started in 2006 was worked inside Lemon, and that record is what trained the models the platform runs on.
Deliberately excluded
- It is a proportion of a findings set. It is not a score, not an accuracy rate and not a comparison of two products.
- It is scoped to the platform rather than to any single one of the eleven coverage classes, LLM applications included.
- Turnaround, where it is quoted on this page, is an observed median of one to three business days on the fully autonomous model only. No equivalent figure exists for expert-verified or human-led work, so none is stated.
- Physical, hardware and wireless testing, which are out of scope for B-52 in every class and on both readings of the term.
Deciding
Four steps, in the order they actually get answered
Only the first one is ambiguous. Everything after it is an ordinary scoping conversation, and it goes faster for having had the first one.
-
Step one
Say which side of the test the AI is on
Tester or target. One clause in the request, and it decides the access you have to arrange, the people who need to be in the room and the shape of the report at the end of it.
-
Step two
Name the coverage class
Eleven exist, and naming one converts an ambiguous phrase into a scope. Only LLM applications has a model as its target; the other ten have your software, your estate or your people.
-
Step three
Choose whose signature the report carries
All three delivery models cover all eleven classes. Fully autonomous returns the same coverage and the same proof per finding without an auditor signature; expert-verified and human-led carry the verification of a senior Security Brigade auditor, and Security Brigade has been CERT-In empanelled since 2008.
-
Step four
Decide how it starts
The self-serve card flow reaches five application classes: web, mobile, API, thick client and secure code review. LLM applications is not among them, so an engagement whose target is a model begins with a scoping call rather than a card — which is a practical difference between the two readings, and worth knowing before you go looking for a button.