Skip to main content
Method · The sixth phase

The report is finished. That is exactly when it gets checked

PTES and NIST SP 800-115 both close their account at reporting, and they are right to: they are describing how a test is conducted. B-52 runs one more phase after reporting — a quality gate on the finished report, which can send a finding back to the phase that produced it before that report reaches you. This is the argument for putting it there.

Yash K · · How it is done · about 12 min

The artefact, not the work

The report is the only part of a test that outlives it

Everything before reporting is work. The report is the thing that gets opened, forwarded, filed and argued with — which is the case for checking the report itself, and not only the work behind it.

A penetration test produces two kinds of output. There is the work — the surface that was mapped, the test cases derived from it, the candidates raised against them, the exploits attempted against those — and there is the report, which is what all of that gets compressed into. Only one of the two leaves the engagement. Your engineers do not watch the run; they open a document and a dashboard, and every decision taken afterwards is taken from what those two say. A finding nobody can reproduce from its own write-up will be treated as noise whether or not it was real when it was found. A severity nobody can derive from the evidence printed beside it becomes a negotiation with whoever owns the release. Neither of those is a testing failure and neither is visible from inside the testing. That is why the sixth B-52 phase sits where it does: the quality gate runs on the finished report, before that report reaches you, and it can send a finding back to the phase that produced it rather than annotate it where it stands.

Read against what is published

PTES divides the work into seven. NIST SP 800-115 divides it into four.

Both are documents about how a test is conducted, and both close their account at reporting. That is a scoping decision rather than an omission, and the difference is worth being precise about before making anything of it.

DocumentHow it divides the workWhere its account closes
PTES Seven phases, running from pre-engagement interactions through to reporting. It is written for the people conducting the test, and the conduct of the test is its subject. At reporting. There is no counterpart in it to a gate that reads the finished report, which is what our own standards register records.
Seven against six is two decompositions of the same work, not a renaming. Pre-engagement sits before our six; post-exploitation folds into our exploitation phase, behind the second of the three approval gates.
NIST SP 800-115 Four phases — planning, discovery, attack and reporting. It is a technical guide to information security testing and assessment, addressed to the team carrying the assessment out. At reporting, and for the same reason. What it governs is assessment activity, and a check on a deliverable is a different kind of question.
Final in September 2008, and neither withdrawn nor superseded. It is cited at that edition here and everywhere else on this site.
B-52 Six phases, and the sixth is a quality gate that runs on the finished report before it reaches you. After reporting. A finding that does not hold goes back to the phase that produced it, and the report is corrected rather than annotated.
The first five will be familiar from both documents above, and this article does not restate them. The sixth is the one worth arguing about.

PTES

How it divides the work
Seven phases, running from pre-engagement interactions through to reporting. It is written for the people conducting the test, and the conduct of the test is its subject.
Where its account closes
At reporting. There is no counterpart in it to a gate that reads the finished report, which is what our own standards register records.

Seven against six is two decompositions of the same work, not a renaming. Pre-engagement sits before our six; post-exploitation folds into our exploitation phase, behind the second of the three approval gates.

NIST SP 800-115

How it divides the work
Four phases — planning, discovery, attack and reporting. It is a technical guide to information security testing and assessment, addressed to the team carrying the assessment out.
Where its account closes
At reporting, and for the same reason. What it governs is assessment activity, and a check on a deliverable is a different kind of question.

Final in September 2008, and neither withdrawn nor superseded. It is cited at that edition here and everywhere else on this site.

B-52

How it divides the work
Six phases, and the sixth is a quality gate that runs on the finished report before it reaches you.
Where its account closes
After reporting. A finding that does not hold goes back to the phase that produced it, and the report is corrected rather than annotated.

The first five will be familiar from both documents above, and this article does not restate them. The sixth is the one worth arguing about.

What the gate reads for

Three ways a report goes wrong while the testing behind it was sound

None of these is a testing failure. Every one of them is a reporting failure, which is precisely why none of them is visible to a check that runs before the report exists.

01 Reproduction

A finding nobody can reproduce from its own write-up

What it looks like
The exploit worked. The write-up leaves out the header that made it work, or the order the two requests had to go in, or which role the session belonged to. Everything in the record is true, and the record is not sufficient.
Why the run behind it looks clean
Nothing upstream failed. The candidate was raised, the exploit was proved, the evidence was captured. The loss happened at the point where evidence was turned into prose, which is the one step no earlier phase gate is positioned to inspect.
What the gate does with it
A finding that cannot be reproduced from its own record is not yet a finding. It goes back to the phase that produced it, and what returns is a record somebody who was not there can work from.
02 Severity

A severity that does not follow from the evidence printed beside it

What it looks like
The CVSS v4.0 vector describes one attack and the write-up describes another — an unauthenticated vector recorded against a path that needed a session, or an impact asserted at a level the proof does not actually reach.
Why it matters more than it reads
A severity is not an adjective on a page. It orders the queue your engineers work through, and it is the number a build gate compares against the threshold you set. Getting it wrong stops a release that should have shipped, or passes one that should not have.
What the gate does with it
The vector is read against the evidence attached to that finding, rather than against a general sense of how serious this class of issue usually is. A severity that does not follow from its own proof is one somebody will argue with, and the argument belongs before the report goes out.
03 The record

A finding that arrives short of what a finding has to carry

What it looks like
Every B-52 finding carries the request as it was sent, the response as it came back, the steps that reproduce it, a CVSS v4.0 vector and a CWE. A finding missing one of those five is a finding that was written up in a hurry.
Why nothing upstream catches it
A phase gate can assert that the work it governs was done properly. It cannot assert that the eventual write-up carried that work across, because while it is running there is no write-up to inspect.
What the gate does with it
The check is made against the finished report — the artefact in the form you would read it — and not against the run that produced it. That is the whole of the difference between a sixth gate and a fourth one.

Position

Why a gate placed earlier cannot do this job

A quality gate can only check what exists at the moment it runs. Move ours to fourth and the thing it was built to check has not been written yet.

Put a gate between exploitation and reporting and it can do one genuinely useful thing: hold work back until the work is sound. That is worth having, and it is close to what the exit condition on each of the earlier phases already does — nothing leaves scanning as a finding, and nothing survives exploitation on a guess. What a gate in that position cannot do is read the report, because the report does not exist while it is running. It can approve a set of properly proven findings and be followed immediately by a document that describes one of them badly. The cost of putting the gate last is real: it sends finished work back rather than unfinished work, and rework at the end of a run is more expensive than rework in the middle of one. The alternative is cheaper to operate and worse for you, because it moves the moment a bad write-up is discovered from before you receive the report to after it — and after it, the discovery is made by the engineer who could not reproduce it.

The gate, as conditions

What has to hold before a report leaves

Each of these is asserted against the finished report rather than against the run behind it. The last row is what makes this a gate rather than a review.

What has to hold before a report leaves
StateWhat it meansWhat follows
Reproducible from its own record The steps in the write-up, followed by somebody who was not there and cannot see the run, reach the same result. The request, the response and the order they went in are all present in the record itself. The finding stands and goes out.
Severity follows from the evidence The CVSS v4.0 vector agrees with what the attached proof shows — the access the attack needed, the interaction it required, and the impact it actually reached. The finding stands and goes out.
The record carries all five parts Request, response, steps to reproduce, CVSS v4.0 vector and CWE. All five, on every finding, in every coverage class and in all three delivery models. The finding stands and goes out.
Any of the three does not hold Terminal The finding does not survive as written. It is not downgraded to a note, and it does not go out with a caveat attached explaining why the evidence is thinner than the claim. Back to the phase that produced it. The report is corrected rather than annotated, and then it is read again.
Key
  • Holds against the finished report, and goes out
  • Sent back to the phase that produced it, and the report is rewritten
  • TerminalNo state follows this one

Downstream

What reads the report once it leaves — and what produced it

The gate sits where it does because of this list. Four of these read the report rather than the run, and each of them is a place where a weak write-up costs more than the finding behind it was worth.

Engineering

An engineer reproducing it on a Tuesday

The person fixing it was not in the run and has no way back into it. They have the write-up and whatever is attached to it. If the steps do not work from that alone, the finding costs more to argue about than it would have cost to fix.

Pipeline

A build gate comparing one number against your threshold

B-52 runs in four continuous-integration systems in production — GitHub Actions, GitLab CI, Jenkins and Azure DevOps — and a result can fail a build against a severity threshold you set. At that point a severity has stopped being a description and become a decision taken automatically about a release.

Procurement

A file somebody will be questioned on

A report that goes into an audit file, an RFP response or a vendor questionnaire gets read by somebody looking for the seam between what was tested and what was claimed. The method is published as a document for exactly that reader.

Standards

The standard each finding cites

A CVSS v4.0 vector and a CWE identifier are checkable by anybody who cares to check them, which is the reason for citing them at all. The register lists every standard this site cites, at the edition it was read at and with the date it was read.

The run behind it

And the five phases that produced the thing being checked

Five phases run before a report exists, each closing before the next opens, with three actions inside exploitation that stop the run for your written approval. The gate above is the sixth of those phases and the only one that reads an output instead of producing one.

Across the three models

The gate does not move when the delivery model changes

All three delivery models cover all eleven coverage classes and run the same six phases. What changes between them is whose signature the report carries.

In the fully autonomous model you sign off the scope and nothing further is asked of you except at the three approval gates inside exploitation. The report is produced, the gate reads it, and the observed median from scope sign-off to the report reaching you is one to three business days — a median taken from real runs rather than a service level, and one that has the gate inside it, since the gate runs before the report reaches you. In the autonomous expert-verified model a senior Security Brigade auditor verifies every finding before anything reaches you; in the human-led model a Security Brigade team works the engagement with the platform underneath it, and Security Brigade has been CERT-In empanelled since 2008. What none of that changes is the ordering. The report is written, then it is read, and what goes out is what survived being read.

The six phases in full, and where the gate sits in them

The platform page walks all six phases with the exit condition each one has to satisfy, and the table under it shows where a person can act in each of the three delivery models. The methodology document states the same method in the form a procurement reader can attach to a file and be questioned on.