Penetration testing is a technique commonly practiced by computer security specialists, but its power is limited—especially when it comes to software.
Since the days of SATAN (the first network security port scanner), computer security specialists have honed penetration testing into a powerful art. Ironically, SATAN's authors Dan Farmer and Wietse Venema's approach—improving the security of your site by breaking into it using automated tools—got Farmer summarily fired from Silicon Graphics in 1996; 10 years later, any sysadmin not regularly using a network port-scanning tool (think Nessus) is ripe for dismissal. We've come a long way in a decade.
Penetration testing certainly has demonstrated its value as a diagnostic exercise for network security. The reason penetration testing works so well for network security is that it focuses on configurations of fairly well-understood systems. That is, networks should be configured certain ways, firewalls should block certain ports, common applications such as sendmail need to be properly patched up, and operating systems on servers should be up to date. All of these essential security practices can be probed using network penetration testing.
Enter the Badness-ometer
The problem with diagnostic exercises is that they're diagnostic exercises. That is, a penetration test can tell you whether your system can be compromised in a straightforward manner (often using canned, black-box probes), but it can't tell you that you're completely secure. Both SATAN (www.fish2 .com/satan/) and Nessus (www.nessus .org) use very straightforward probes. However, penetration testing like this simply demonstrates a negative. Whether a problem found through penetration testing is easily corrected depends entirely on the nature of the problem. It's straightforward enough to tweak firewall rules and bring operating systems up to date, but if a penetration test finds a problem deep inside your software, what are you to do?
When software branches off into nonstandard applications, penetration testing begins to lose its potency. Think about it in terms of testing any arbitrary program for a negative condition. Many "application security testing" tools rely on canned black-box tests to do their business. That is, they try to simulate a "hacker in a box" for testing software security.
The problem with this approach is twofold. First off, if black-box canned tests do find security problems, then you know your software is in really bad shape. However, if you pass all of the black-box tests with flying colors, you really don't know anything about your security posture at all (other than that you passed a bunch of tests); hence, the term "badness-ometer." Badness-ometers register values in the range from "Deep Trouble" to "Who Knows." Note that a badness-ometer is not at all the same thing as a security-ometer. Second, penetration testing is often outsourced to consultants, many of who claim to be "reformed attackers"—well, they say they're reformed. How can you know whether you're getting all of the results? Is knowing two of five ways to hack into your system good or bad? What makes admitted criminals (reformed attackers) worthy of trust?
Diagnosis of a problem with black-box testing often does not result in a clear mitigation plan. So far, that hasn't been a problem with the use of network scanning tools (SATAN derivatives), but it's a big problem with software security.
Embrace the Badness
Then again, it's good to know if you're in deep trouble. If you know, maybe you'll do something about it. For that reason, I hope everyone buys a badness-ometer for their software and applies it today. We're all very likely to be shocked at the state that we're in—a state of untenable software risk. Then maybe we'll turn to building security into our systems from day one.
