Black box, gray box, and white box: three levels of knowledge in a test

The difference between the variants isn’t in how thorough the test is, but in how much of the agreed time goes to finding out information the client already has. One note on spelling: gray box and grey box are the same thing; we use the US form, gray.

Black box makes sense where the goal is to verify what an attacker sees without any access at all: the external perimeter, a public portal, a mail gateway. For an application behind a login, though, a large part of the time is spent obtaining an account and mapping the features, work that gray box settles by handing over credentials.

The argument for black box is that it’s more realistic. It rests on an equation that doesn’t hold: the less the tester knows, the more faithfully they supposedly simulate an attacker. The trouble is the time budget. An attacker can push at one company for months at no cost; the tester has 15 days. Simulating ignorance therefore means buying reconnaissance instead of findings.

• The external perimeter is being tested and the question is what can be seen from there at all.
• The brief includes checking whether anyone catches the attack.
• The application is public and the tester can register an account themselves.
• The client needs to show a counterparty that the test ran without inside information.

Choose black box if you’re testing what is visible from the internet, not what happens after login.

Gray box is the default choice for web applications and APIs because the tester gets an account for every role and can go straight to checking whether it’s possible to climb from one role into another. Authorization flaws, meaning access to someone else’s data after a legitimate login, can’t be tested without an account for every role.

Handing over accounts also solves a practical problem: without them, the tester has no idea that besides the ordinary user the application also has an approver role and a branch manager role. The range of checks the tester goes through is governed by the methodology, not by the level of knowledge; for web that is most often the OWASP Web Security Testing Guide.

Choose gray box if the application has more than one user role and you care that all of them get tested.

White box pays off for applications that the company develops itself and that carry a lot of custom logic: payment flows, limit calculations, integrations with banking interfaces. With the code, the tester also verifies branches that can’t normally be reached from the interface and, for each finding, points straight to the place that needs fixing.

The worry that handing over code gives something away to the tester misses the point. Code doesn’t raise the severity of a finding; it only shortens the path to it. What the tester may and may not do with it is governed by the NDA and the rules of engagement anyway, not by keeping the repository secret.

• You develop the application yourselves or have access to the code through your vendor.
• A substantial part of the risk is in the logic, not the configuration.
• The test is meant to follow up on a code review or on fixes from the previous round.
• The application processes payments, health data, or other sensitive data.

For an application, the level of knowledge means accounts, documentation, and source code. For a network it’s something else: whether the tester knows the topology, IP ranges, and list of systems in advance, or has to find them alone. The same word in an inquiry for a test of an application and its roles and for a test of networks and servers therefore means different preparation each time.

For an internal network there is also a third variable that box terminology doesn’t cover at all: what privileges the tester starts with. Whether they get an ordinary domain account or first have to get onto the network themselves changes the result of the test more than how much documentation they read beforehand.

The list of inputs belongs in the brief, not in the kickoff call. When accounts and test data get sorted out only after the start, the time comes out of testing, regardless of the variant chosen.