How to choose a penetration testing vendor and what to ask

Before you approach vendors, settle three things: what is to be tested, what you want the test to tell you, and which type of test you need. Without that you get proposals that can’t be compared, because each of them assumes something different.

The scope is the list of what is in the test and what is not: specific applications, IP ranges, domains. The goal is the question the test should answer, within the limits of what a test is for and what it is not: whether one application holds up before it goes live, or how far an attacker gets from an ordinary workstation. The type of test then follows on its own. The details you hand over later, to the vendor you pick, that is what to give the tester after the order, are covered in a separate article. This is the phase before that.

A good proposal answers five questions even if you don’t ask them. When you can’t find them in it, ask, and the answers make the difference between vendors clear fast.

• How many days are reserved for the testing itself. Not the total length of the engagement, but the pure time spent on your environment. That is the one number that says something about depth.
• Who will actually run the test. Ask for the person’s name in the proposal and check that the same name will be on the report. A salesperson writes the proposal. A tester should run the test.
• Whether a retest is included. A retest, meaning a repeat check that the fixed findings really are fixed, is billed separately by some vendors. Without it you have nothing documenting that anything is fixed.
• Which methodology the testing follows. A reference to a recognized procedure, not to an in-house secret process, means the result can be compared and repeated.
• References from your sector. Not a client name that is under NDA, but the type of environment and the scope the vendor has worked with.

The best proof of quality is a sample of a real deliverable, not a presentation. Ask the vendor for a sample report before you order. It shows you what a proposal won’t: whether a finding can be reproduced from the description and whether the remediation priority makes sense.

A sample shows the difference between a report and a scanner export. A report carries evidence for each finding, the steps to repeat it, and the impact in your environment. An export is a list of versions and scores from a database that nobody has been through by hand. What such a deliverable looks like and how you recognize a well-run test is described on our service page.

We provide a sample report too. It comes from our own test infrastructure, not from a client engagement, so there is no third-party data in it, and we email it to anyone who asks. Ask other vendors for a sample as well, and if they refuse to show one, that is an answer in itself.

Don’t get lost in the number of acronyms. What matters is who runs the test and whether they hold a hands-on certification, meaning an exam where they had to break into an environment for real, not just answer questions.

The difference between a hands-on exam and a knowledge exam, and which tester certifications mean something, is covered in a separate glossary entry. To choose well you only need to know that the number of acronyms says nothing about quality and that you ask about the qualifications of the person who will test, not of the company as a whole.

Before you sign, go through whether the contract covers six things. A missing point is harder to make up for once the test is running.

• Written authorization to test from someone who can give it. Without it, this is unauthorized access, not a test.
• The exact scope: what is in the test and what is not, so it can’t be widened or narrowed afterwards.
• Rules of engagement: time windows, contact people, the escalation path, prohibited techniques.
• An NDA or a confidentiality clause, because the vendor will see your weak spots.
• A retest of the fixed findings, so it is clear whether it is in the price or costs extra.
• Handover of all findings, not just a summary. You want the technical section with the evidence as well, not a presentation for management on its own.

Some signals give away a weak proposal before you open it. Four of them are worth watching for.

• A scanner export passed off as a report: thousands of rows, no documented evidence of exploitation, no remediation priority.
• A price without a scope: a flat fee set before the vendor knows how much there is to test.
• No sample deliverable: a vendor who won’t show a sample report either doesn’t have a good one or won’t stand behind it.
• No clarity on who tests: a proposal with no tester’s name, or a promise of a senior team that then appears nowhere in the report.

With a price set without a scope, it helps to know what the scope is actually driven by. We cover how it works and what goes into the scope and the price separately. Choosing a vendor isn’t about the lowest price. It is about whether you get a deliverable you can act on.

By who runs the test, how they run it, and what you get out of it. Ask for the tester’s name and their hands-on certification, ask about the methodology and whether a retest is in the price, and ask for a sample report. References from your sector help. The lowest price is not a criterion. What matters is the quality of the deliverable.

Five things: how many days are reserved for the testing itself, who will actually run the test and whether their name will be on the report, whether a retest is included, which methodology is followed, and whether they will show you a sample report up front. The answers make the difference between vendors clear fast.

By the deliverable. A scanner export is a list of versions and scores from a database with no documented exploitation: thousands of rows nobody verified. A real report carries evidence for the finding, the steps to repeat it, and the impact in your environment. A price set without knowledge of the scope, and a refusal to show a sample up front, are warning signs too.