The Differences Between Audits, Security Assessments and Penetration Tests

We live and work in a world where malicious activity and cyber crime run rampant.  While online fraud and data theft have existed since the origin of the internet, never before has it seemed to be as pervasive as it is today.  Every day there seems to be a new headline about ‘XYZ Company’ coming under attack or having a massive data breach occur.  On top of that, in the United States, cyber crime has reached new heights in our consciousness given the current political climate and questions about interference with the last election. 

Needless to say, public awareness around information security is here to stay and businesses are, mostly, on high alert.  With that in mind, it seemed fitting to put some thoughts together about what this all means and discuss some of the things that are being done by businesses and other organizations to proactively bolster themselves against emerging threats.  Specifically, the aim of this post is to shed some light on security assessments and penetration testing.  One of the absolute best ways to measure one’s security posture against threats and attacks is to conduct testing that emulates those threats and attacks in a real-world, yet safe and controlled, manner.

Therein lies the problem, however, as there is a great deal of confusion and misrepresentation in the market around security assessments and penetration tests.  Misused and/or misinterpreted terminology is a common pitfall that I frequently encounter when addressing these topics with clients, prospects, partners, and other folks in this shared space.

The importance and value of conducting penetration tests as a part of an organization’s regular business process seems to now be well understood and accepted as a best practice, contrary to as recently as a few years ago.  This is great news, but the problem that still exists is that many, if not most, still do not clearly understand what it is that they’re asking for when they say something as seemingly straightforward as “I want a penetration test”.  Traditionally, there are three stages that constitute a penetration test: scanning, assessing and penetrating.

Why Audits Are Not Penetration Tests

One of the biggest misconceptions is that conducting regular ‘audits’ as a part of a certification process is the same as having a security assessment performed.  Simply put, that is false.  An audit is not a penetration test.  Rather, an audit is a check-box activity that is designed to go through a list of regulatory items tied to compliance to ensure that ‘things’ are in place, be it technology, internal processes, organizational structures, etc.  Audits may not even include any stages of a penetration test.

The problem with that is that while audits can help organizations identify certain gaps in those areas, it does not properly test the effectiveness of those components.  Checking a box that says you have a firewall is not the same as testing to make sure that said firewall has been removed from its package, been properly configured, and updated with patches and other relevant changes that need to be made on an ongoing basis.

Compliance certainly has its place within the framework of a strong organizational security posture, however, compliance alone is not enough.  The whole focus of security assessments and penetration tests is to demonstrate what the real “bad guys” out there could do to any given organization if a motivated and skilled attacker were to spend time on a dedicated effort to compromise their target.

To borrow the cliché of thinking inside or outside the box, an audit mentality is absolutely akin to thinking inside the box and following a predetermined set of rules and guidelines for protection.  Passing an audit and becoming compliant should only be seen as a starting point though because the global community of malicious actors, hacktivists and state-sponsored cyber criminals are very well aware of those predetermined guidelines.  As such, they know how to think outside the box and work around those guidelines to get to what they want.  Many high profile businesses that have had their names in the headlines over the past few years passed their respective audits and were compliant with all of their requisite certifications.

Penetration Tests are Not Commodities

In addition to confusing a proper security assessment engagement with an audit, there seems to be another common misconception in that penetration testing has become a commodity.  While I agree that some aspects of the process are reliant on somewhat commoditized technology, I disagree strongly with the notion that all penetration tests are created equally.

If the idea is to emulate a real world attack to gauge the strength of an organization’s security, then it logically follows that the skill sets of the testers providing those professional services to their clients will serve as the differentiators of value and effectiveness.  I like to make the comparison of professional penetration testers to healthcare providers and physicians.

When selecting a doctor to perform a surgery, most individuals will go through an exhaustive evaluation process to make sure that the skill sets and qualifications of that doctor match with what needs to be done.  I would assume that many people put significantly more time and effort into that sort of a decision than they would in purchasing a hammer or a wrench.

It may be an oversimplification, but the idea is that when dealing with something as important as one’s health, people want to make sure that they’re getting the proper care based on the skillsets and expertise of the individuals performing the task at hand.  Whereas a hammer, for example, simply needs to strike a nail hard enough to send it into or through its target.  The idea that not all doctors are created equally mirrors my thoughts that not all penetration testers are equal.  It is important to keep that in mind when evaluating your options.

What Constitutes a Penetration Test

To go deeper into that idea, I’ll spend a little time discussing the process that goes into a proper penetration test.  This seems to be an area where a lot of confusion originates.  As previously mentioned, there are three stages that go into a traditional penetration test; scanning, assessing, and exploitation.

Scanning is a process that leverages automated scanning tools to conduct a preliminary evaluation of an organization’s perimeter.  I liken this to tying your shoes before you go for a run.  It is a necessary step in the process but it alone is not sufficient for running a race – you’ll still need to manually put one foot in front of the other to cover the distance.  Scanning is useful in that it can quickly identify potential vulnerabilities that exist within a network, however there is still room for false positives and other incorrect readings.

The second stage of a traditional penetration test is an assessment.  During the assessment stage, a tester will review the preliminary findings of an automated scan and manually validate those findings by trying to compromise the targets that were initially identified as being potentially vulnerable.  In doing this, the singular goal is to determine that there is an issue that could be exploited for that given target but go no further.  This would be like trying to pick a lock to prove that a ‘secure’ door could be opened, but going no further after successfully picking the lock.

Last, but certainly not least, is the exploitation stage.  This portion of a penetration test carries the most value in that it is when an organization can quantify the risk and potential damage that could be t committed by having an identified vulnerability exploited by a malicious individual or group.  During this stage, testers will essentially demonstrate, in a safe and controlled manner, how far, deep, and wide any given wormhole would go from its initial stage of compromise.


An unfortunate reality of our world is that there is no such thing as an infallible level of security.  If one is going to be ‘connected’ to the outside world, all individuals and organizations must operate with a certain level of risk or vulnerability.  To that end, the best starting point for identifying an acceptable level of risk is to understand the likelihood of compromise and how much effort it would take to address the targets of opportunity within any given environment.

This comes back to the original topic of what it means when someone says “I want a penetration test”.  To me, if someone is looking for a true penetration test, they will spend time evaluating the methodology of potential service providers and the skill sets of the professional testers employed by those providers.

The first thing I try to do when meeting with people regarding penetration testing is to clearly understand how they define the meaning of ‘penetration test’.  Once a common ground of understanding is reached, it becomes possible to mutually identify the task at hand and put a plan in place to adequately address the organization’s goals, needs and desired outcomes.

An ongoing practice of proactive security measures is the only way to give oneself a chance at defending against attackers.  Too many organizations make it way too easy for attackers to take what they want.  If you are tasked with protecting your organization’s most critical assets and work in a highly regulated industry, compliance is a good starting point.  Just be sure not to stop there.  Explore your options for security assessment and penetration testing providers and make sure whomever it is you decide to work with can clearly define to you what it is they do and how they do it.