One of the talks I most enjoyed at the 2013 BsidesLV conference was Ed Bellis and Michael Roytman’s “Vulnerability & Exploit Trends:A Deep Look Inside The Data”. Ed and Mike performed a data-driven analysis of the problems presented when vulnerabilities are prioritized based solely upon their CVSS ratings. The Risk I/O argued that using the presence of vulnerabilities in attack frameworks such as Metasploit and the prevalence of vulnerabilities in publicly reported breach events has a much better return on investment than starting at CVSS base scores of 10 and working down a list of issues.
Background and Motivation
My organization has done a lot to improve the focus of our vulnerability remediation efforts. We make heavy use of commercial tools such as RedSeal and custom analytic and reporting tools (including visualization packages like Tableau), and we’ve made great strides in identifying what’s important and what’s not. Despite this, the universal availability of CVSS makes it hard to avoid and it is an important element in our current ranking of the exploitability of a vulnerability. After seeing Ed and Michael demonstrate that alternative approaches can be readily implemented with currently available data, I wanted to jump in and give this a try myself.
Tools, Techniques, and Procedures
The first challenge was to find a reliable source of parsable and high quality mappings between vulnerabilities and public exploits. This turned out to be more than difficult than we had expected, but we were eventually able to test a VulnDBfeed from the folks at Risk Based Security. With access to the API, I was able to write a quick and dirty Python script that pulls down JSON formatted dumps of a sample range of vulnerabilities. I then leveraged techniques picked up from the Verizon RISK team to import the JSON into MongoDB. From there I dove into the aggregation framework to create a CSV with the OSVDB ID, Metasploit reference URL, and the CVE ID. With this source CSV in hand, it was quick work to pull this data into Tableau, map it against a population of hosts that I was already tracking and add a dimension of Metasploitability to each of the CVE identifiers.
How Did It Turn Out?
Of the classic severe vulnerabilities (CVSS 7+) that had been on our radar for tracking, less than 4% were associated to publicly available attack code. On the other hand, attack code was available for several vulnerabilities in the less severe CVSS buckets. With the ability to model our network attack paths in RedSeal, this new data dimension looks very compelling to adjust particular vulnerabilities up or down in our work queue for remediation.
I now have a much more robust picture of the actual seriousness of existing vulnerabilities. This will allow me to answer questions such as which vulnerabilities matter most and which systems are possible vectors for network attacks, and this in turn will enable me to work more effectively with my organization to address these issues. There is still quite a bit of work to be done before we can recommend kicking CVSSv2 completely to the curb, but I plan to explore this further over the coming year.
About the Author (or, This Really Isn’t That Hard)
I am a rank beginner with both python and MongoDB. Jay Jacobs provided invaluable assistance during my first forays into MongoDB. From starting to work with the API to authoring this blog post has taken me about 20 working hours, with most of that time spent learning how to work with OAuth and JSON in Python and with the aggregation framework in MongoDB.