5 Questions to ask about Vulnerability Prioritization Solutions


Over the past three years, the number of software vulnerability disclosures have surged.  In a typical month, there are over 1,000 new disclosures from NIST.  Further, the standard ranking system provided by NIST, the CVSS scoring, does not aide significantly in prioritizing software vulnerabilities – for example over half are rated as high or critical.  Also worthy of note, several scientific studies have shown that the CVSS scoring is not predictive of exploitation in the wild. Due to these challenges, CISO’s and their vulnerability management teams have sought solutions to improve on prioritizing software vulnerabilities.  Here are 5 questions to ask when choosing a vulnerability prioritization solution.


5 criteria (1)


1. Does the solution include threat intelligence? 

Threat intelligence, especially information coming from places hackers discuss, sell, and share exploits, is critical in proper prioritization.  Typically, this type of information comes from the deepweb, darkweb, social media, open sources (i.e. exploit repositories), and other sources.  Many solutions will provide only provide information on very-well established exploits (i.e. those in Metasploit) or freely available exploit repositories (i.e. ExploitDB).  While these represent a good start, they only provide a small piece of the picture as to where the hacker interest lies.  For example, peer-reviewed research has shown that about half of vulnerability discussions on the darkweb occur before exploitation.1 


2. Does the solution support my workflow and provide an API? 

Many vendors advertising vulnerability prioritization solutions are attempting to lock the user into a given platform – whether it be a vulnerability scanner, orchestration tool, or a risk management platform.  While this has advantages to users of those existing platforms, it greatly limits flexibility.  For example, if you can prioritize vulnerabilities by threat intelligence, it might make sense to ingest that into your SIEM to triage alerts.  If you utilize an external scanner that also reports vulnerabilities, you might want to include those results in the prisonization scheme.  Further, if you are dealing with dev-sec-ops teams, you might want them to use the vulnerability prioritization information before they deploy new systems.  The best way to address the issue of flexibility is to ensure the vendor has API’s and support for various different workflows.  This also allows the CISO to evolve his security program outside the confines of a single platform.  For example, if you pick a vulnerability pyritization solution focused on traditional IT infrastructure and later you need to support the CI/CD process, you should have that flexibility and not be forced to buy (and justify) a duplicate solution. 

3. Does the solution quantify threat? 

The combination of intelligence and solid API’s can allow for easy integrations with a variety of systems – not just vulnerability management, but systems for application security, SIEM, and others.  However, supplying a dump of intelligence through an API will likely be of limited value.  When dealing with a large number of vulnerabilities, and an even larger amount of intelligence sources, it becomes important to not just give data, but rank the vulnerabilities based on the likelihood of exploitation.  

For example, a vendor can provide a lot of data is provided about vulnerabilities – down to every retweet of the original disclosure.  This will lead to large proportion of vulnerabilities with associated “intelligence” – even if it is some commentator re-Tweeting NIST’s latest disclosure.  As a result, there will be not only information overload, but most of that information will not valuable (only about 3% of CVE’s get exploited in the wild).  It’s important that the vendor measure the quality of the data and determine how likely a threat is to act based on that data – as opposed to only providing such information. 

Along those lines, there are some solutions that will simply ingest intelligence from a traditional provider.  However, the same problem arrives – as the intelligence is not being evaluated for quality or the likelihood of a hacker action.  This approach often leads to high false positive rates. 

4. What level of triage does the solution provide?

One of the major failures of the NIST CVSS scoring system is that it simply does not provide a great deal of triage.  The common practice is to patch “highs and criticals” which still leave you with a lot of vulnerabilities.  Many products use prisonization schemes that are directly based on CVSS, and as a result don’t provide very much triage.  Further, the method of scoring, should put the riskiest vulnerabilities first – ones that are most likely to be exploited.  Several studies have shown that scores provided by CVSS (and their derivatives) generally do not trend this way.  Its good to understand what fraction of vulnerabilities the prioritization system recommends for action, and what is the level of accuracy, in terms of predicting exploits, that it provides. 

5. Does the solution base results on exploits in the wild?

Finally, the key really is to get ahead of hackers, so the method of prioritization should be based on predicting what vulnerabilities hackers will actually use in the wild.  Several prioritization schemes are not predictive at all – but rather look at things like volume of hacker discussions – which does not strongly correlate with exploitation.  Others look at availability of proof-of-concept (POC) code alone – but in reality only a fraction of such POC’s get exploited and there are many exploits with no preceding POC.  Finally, there are machine learning methods that are not trained on exploits in the wild, but other criteria – this is mainly due to the fact that predicting exploitation is notoriously difficult.  For example, we have seen methods that predict POC’s – however this leads to compounding errors (as you are predicting something that does not cover all exploits) which will make accuracy much worse than claimed by the vendor (often to the point of being no better than random guess). 

Parting thoughts 

Prioritizing software vulnerabilities is a very important task, and with the large fraction of vulnerabilities disclosed, the cost of patching, and organizations continuing to suffer breaches, it will continue in importance.  However, there are many approaches – and much of what is being advertised in the industry was previously proven to not work in rigorous studies.  We hope this blog helps in understanding the potential pitfalls with such approaches.