Security audit and penetration test: when you need which

The table comes down to one sentence: an audit judges whether security is described and managed, a test judges whether it works. Neither can answer for the other.

The word audit covers three different services, and that is where misunderstandings between client and vendor start. Before you agree on a price with a vendor, say which one you mean, because they differ in what they deliver and in who is allowed to perform them.

• Certification audit against a standard: assesses conformity with the standard and ends with a certificate or a nonconformity.
• Technical configuration audit: checks system settings against a recommended state.
• Internal process audit: verifies whether people follow their own policies and whether there are records to show it.

A penetration test is none of these. It is closest to a technical configuration audit, but differs in that it doesn’t just go through the settings; it looks for a way through them. A strong configuration audit doesn’t have to follow a checklist, either: it can be built on offensive experience, asking how an attacker would abuse each setting, like a technical configuration audit from an offensive point of view. The difference from a test remains: an audit looks at configurations together with you and with your consent, a test breaks through them on its own. If you’re dealing with the first of the three meanings, certification, we cover separately how a penetration test relates to ISO/IEC 27001 certification.

An audit reaches things that can’t be read from operations. Who approves access, whether it’s revoked when an employee leaves, how changes are managed, who is responsible for backups, and whether a restore has ever been tried. A tester won’t find this out, because they look at the system, not at the organization around it.

The second thing is completeness. An auditor goes through the whole environment against a list of measures, whereas a test sticks to the agreed scope and deliberately ignores everything outside it. When you need to know where you have nothing at all, that is a job for an audit.

A test shows the gap between documentation and reality. A policy may correctly require multi-factor authentication, and yet there is one interface where it was forgotten. In the audit report the measure is in place; in the test it is a way in.

The second thing is impact. An audit states a nonconformity; a test documents what that nonconformity means: that the customer database can be downloaded through it, or an administrator account obtained. When you need practical verification of whether someone can get in, a compliance report alone won’t replace it.

You need an audit when you show the deliverable to someone outside the company, and a penetration test when it is meant to serve your own IT. For a certification body, a customer, or an insurer, start with an audit, because it speaks their language. For deciding what to fix first, start with a test.

• Choose an audit if you’re preparing for certification or a business partner has asked you for evidence.
• Choose an audit if you have no idea where you stand and need an overview across the whole organization.
• Choose a test if you run an application or network reachable from the internet and want to know the real risk.
• Choose a test if you’ve fundamentally changed the architecture or added an integration since the last check.

In a compliance project the two almost always meet. For technical measures, the auditor doesn’t want to hear that the measures are in place. The auditor wants to see evidence, and the most convincing evidence is a test report with a date, scope, methodology, and verification that the findings were fixed.

The sensible order is an audit first, so you know everything that is missing, then a test aimed at the part where technical resilience is at stake. For new systems, it also pays to have the design assessed before anything is built, because a fix on paper is cheaper than a fix in production.