Vulnerability and Security Configuration Assessment Solutions Comparison

Bottom Line: Vulnerability assessment tools play a critical role in enterprise vulnerability management as well as broader information security. The tools are being expanded toward in-depth security configuration assessment, large-scale vulnerability prioritization, and security assessment of new cloud, mobile and virtualization environments. All large enterprises should use such tools, procured from vendors with extensive and lengthy experience at enterprise scale.

Context: Both operating systems and application software deployed in production will always have some vulnerabilities, even if secure software development life cycle (SDLC) practices become much more widely employed. In general, the same applies for custom software even more so than off-the-shelf software. As a result, the need to discover and fix vulnerabilities on production systems will always be there. Today's vulnerabilities include both development stage software weaknesses and operation stage configuration weaknesses. Newer mobile, virtual and cloud environments have the same issues in addition to their own blend of challenges. Vulnerability management relies heavily on vulnerability assessment (VA) tools but also uses tools from adjacent markets.

Take-Aways:

• VA tools are products, but vulnerability management (VM) involves people, process and technology. VA tools are a critical part of VM, but they're not the whole of it.
• Assessment and management products are separate, just like most of the teams that perform assessment are separate from those that manage operations. In many organizations, the remediation task stays with the operations team, and vulnerability assessment goes to the security team.
• The vulnerability assessment market comprises a clear set of enterprise-ready products that compete for enterprise security budgets. The vendors mostly compete with each other and not with the "long tail" of the remaining vulnerability assessment players.
• The barrier to entry into this core group of vendors is significant: Vendors' multiyear experience with (and depth of support for) successful scanning of large environments has led to development of features essential for enterprises, and it is likely impossible for other vendors to "jump ahead" without spending the same time learning the same lessons.
• Key VA market and technology trends are:
  • Dealing with huge numbers of vulnerabilities is still an issue.
  • Organizations are seeing the limitations of network vulnerability scanners versus custom application vulnerabilities.
  • Compliance remains a strong driver for vulnerability assessment as well as remediation.
  • Most vulnerability assessment players have integrated security configuration assessment (SCA) features into their authenticated scanning.
  • Adoption of cloud-based management services for vulnerability scanning is expanding.
  • VA tool output data is used more widely.
• Recommendations:
  • Use a dedicated product for audit/assessment; in most situations, your system management tools should not be used for vulnerability assessment (a rare exception is when an agent-based management product supports the required platform while the remote vulnerability assessment product does not).
  • For large enterprise deployment, choose the vendors that can demonstrate years of enterprise experience in large deployments.
  • Request vendor references from large deployments, similar in size to your organization, that have been running the tool you are considering for at least two years.
  • Plan your VA deployment architecturally and then talk to references about their approaches and their views on your proposed approach.
  • Compare vendors on core capabilities first, but don't stop there. Also compare vendors' approaches to emerging environments such as virtual, cloud and mobile. Investigate vulnerability assessment for new environments now, before they are widespread.
  • Test both VA and SCA capabilities on platforms and applications in your environment in order to check whether the depth of such assessment is adequate.
  • Evaluate products not only on the speed and accuracy of VA scans, but also on management and administration features. For remotely deployed components, carefully verify the extent of centralized management.
  • If ease of management of the highly distributed VA tool is a key priority, look into end-to-end (application and operating system) cloud/hosted management tools.

Conclusion: Continuing compliance concerns are driving the VA market, but risk reduction and internal policy compliance drive VA as well.

Audit-only products have separated from management products to provide a means for security groups to independently assess system security. The VA market will continue to evolve along the audit-only path for the foreseeable future. The vendors discussed in this comparison assessment will likely retain their leadership and even expand the gap, using their more than decade-long experience with large enterprises. Large enterprises should consider such tools and procure them from vendors with an established track record for support.

When the first commercial operating systems and software applications appeared, vulnerabilities were born. In later years, broad Internet accessibility and proliferating compute resources made finding and remediating vulnerabilities a key priority. Production software will likely continue to have vulnerabilities for the foreseeable future. In general, the same is even truer for custom software, which is developed by internal teams or partners, than it is for the off-the-shelf variety.

As a result, the need to find and fix vulnerabilities on production systems will persist as long as vulnerable systems are exposed to threats and there are people motivated to take advantage. Such vulnerabilities include both development stage software weaknesses and operation stage configuration weaknesses. Newer environments, including mobile, virtual and cloud, are subject to the same rules.

This assessment of VA tools focuses on the market landscape and vendors' ability to help manage known, reported vulnerabilities in commercial software and operational software configuration weaknesses. It also focuses on the tools' emerging capabilities, such as large-scale vulnerability prioritization and coverage for new IT environments. Additional details on the underlying technology are covered in the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology."

Introduction

No matter how hard you try, you cannot go to a store and purchase "vulnerability management." Gartner defines vulnerability management as "the key process for finding and remediating security weaknesses before they are exploited." As part of this process policy definition, assessment, shielding, mitigation and monitoring are required. Security processes, unlike appliances, software and services, cannot be acquired in exchange for cash. They can only be established by an organization and then mature to an appropriate level.

VM encompasses the whole life cycle of processes and multiple technologies aimed at reducing IT vulnerabilities and mitigating their impacts. In order to enable and support the process, the organization will rely on a multitude of tools, both within and outside the domain of information security technologies. VA covers tools for finding known vulnerabilities and configuration weaknesses on computing resources, such as servers, desktops, mobile computers, and other networked devices as well as related workflow processes, such as vulnerability prioritization and analysis.

In this assessment, we adopt a narrower definition of VA by confining it to tools that look for known, reported vulnerabilities across systems (for example, those assigned a Common Vulnerabilities and Exposures [CVE] ID). This makes it possible to separately define SCA as looking for unsecure configurations of systems and applications. This separation is helpful because, whereas vulnerability scanning capabilities are approaching commodity status, SCA capabilities still vary widely across the tools and are farther from mature commodity status. A combined VA/SCA label will be used to indicate broad vulnerability scanning capabilities as present in many tools on the market today.

Broader Vulnerability Management Context

In order to understand and intelligently compare VA/SCA tools and measure their contribution to an organization's overall security posture, it is essential to shed light on the broader context of vulnerability management.

Source: Gartner (March 2012)

Figure 1 shows a wide range of security tools utilized at various stages of the VM life cycle. Specifically, VA/SCA tools gather the baseline information on assets (and discover the assets on the network). Network intrusion prevention systems (NIPS) or host intrusion prevention systems (HIPS) temporarily shield vulnerabilities, and configuration management tools remediate them. Threat intelligence feeds and security monitoring tools such as security information and event management (SIEM) assist with vulnerability prioritization and ongoing security monitoring. To ultimately provision systems without known vulnerabilities, change must come during application development and initial system provisioning. For an in-depth discussion of the vulnerability management life cycle, review the Gartner research notes referenced in the Recommended Reading section.

VA/SCA tools compose a subcategory of the larger market for security detection, prevention and response products that includes such mainstream technologies as firewalls, intrusion prevention systems (IPSs), anti-malware systems and SIEM. On the other hand, tools that manage — as opposed to assess or audit — the systems are mostly provided by large vendors in security and IT management. For example, the network configuration and change management (NCCM) market is dominated by BMC Software, CA, IBM and Symantec and has very few stand-alone vendors. This hints at one of the key characteristics of the VA tool market — it caters to a separation of duty imperative.

Substantial differences exist between technologies that assess or audit and those that manage or control. This conceptual "separation of duty" is highly visible in the market and across the domain of technologies, appears to be a natural evolution in the market and is beneficial for security. Just as organizations assign different people for audit and management tasks, they often use different tools. In most organizations, the remediation task stays with the network or server operations teams while vulnerability assessment and analysis is performed by the security team.

Simply put, vulnerability assessment is not the technology to "solve" a security problem for an organization. At best, these technologies and processes can only "soften the blow," but not fully protect from vulnerability. After all, one cannot fix all the problems introduced during development at the operation stage; vulnerabilities can only be "managed" and rarely entirely "removed."

Scope

The following customer questions will be tackled in-depth in this assessment and its companion documents, the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology" and "Vulnerability Management":

• How should an organization do vulnerability management for maximum risk reduction?
• How should we prioritize vulnerabilities for remediation?
• What should an organization look for in a VA tool that will work for a global distributed environment?
• How can scanning help achieve and maintain compliance?
• How should we architect scanning for a large environment?
• How should we build a successful VM process using VA/SCA tools?
• Will existing VA/SCA tools work if an organization needs to assess virtual, cloud, Internet Protocol version 6 (IPv6), supervisory control and data acquisition (SCADA), and mobile environments?

This comparative assessment covers vulnerability assessment tools that can be used during the vulnerability management process. However, it does not provide a detailed discussion of patch management, system management, penetration testing, Web application scanning, desktop management, database scanning, code scanning, network configuration and change management, and other tools related to the process. Interested readers can check the Recommended Reading section for many Gartner references to these other topics.

Gartner customers that use vulnerability assessment tools and practice vulnerability management struggle with the following problems, which will be focus areas for this comparative assessment:

• Vulnerability prioritization for remediation
• Large deployment architecture
• Scanning new environments: Virtual, cloud, mobile, and other non-PC assets

VA/SCA vendors will be compared based on core capabilities as well as on the above focus areas. For deeper coverage of technology used for prioritization, new environment assessment as well as large environment architecture, see the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology" document.

Vulnerability Assessment Market

The vulnerability assessment (including security configuration assessment) market is defined to encompass tools that can probe operating systems and production applications for vulnerabilities.

Market Basics

VA/SCA tools are offered by a long list of vendors, with the oldest being in business since the early 1990s. However, Gartner estimates that only five of the vendors earn 80% of the revenue.

Gartner defines the market to consist of tools that:

• Perform active¹ network VA
• Provide vulnerability information and reference multiple vulnerability IDs, including common vulnerabilities and exposures, SANS Top 20, Bugtraq ID and vendor-specific IDs
• Provide remediation guidance
• Focus on the security organization
• Provide asset classification capabilities

The broad VA market includes some tools with niche focus, such as Payment Card Industry Data Security Standard (PCI DSS) compliance. Some of the tools might not even offer SCA capabilities, while others expanded their focus to multiple VA methods, SCA, dynamic application security testing (DAST) and — to a limited extent — database security assessment and other areas.

The key market drivers are:

• Risk reduction
• Regulatory and policy compliance
• System auditing

Despite the prominence of compliance and contrary to popular opinion, VA tools and VM practices are not married to compliance because risk reduction drives a significant percentage of vulnerability assessment projects. In general, VA capabilities are driven more by risk reduction, and SCA capabilities deliver more value for compliance. SCA and asset discovery capabilities are sometimes driven by the need to improve asset management practices:

• Today's compliance driver is much broader than PCI DSS Approved Scanning Vendor (ASV) scanning. Although it is true that PCI DSS has impacted vulnerability scanning significantly since 2005, PCI ASV scanning constitutes a relatively small portion of the market for enterprise VA/SCA vendors. PCI DSS internal scanning, PCI DSS automated control assessment, United States Government Configuration Baseline (USGCB) control assessment (formerly known as Federal Desktop Core Configuration [FDCC]) and the seemingly emergent Health Insurance Portability and Accountability Act (HIPAA) risk and control assessment present well-known examples of compliance-driven VA/SCA. Both VA and SCA capabilities are critical because many regulations prescribe both technical control assessments (which drives SCA) and vulnerability assessment (which drives VA). Some mandates, such as PCI DSS, even mandate that all vulnerabilities with severity scores of "medium" and "high" (or with Common Vulnerability Scoring System [CVSS] scores over 4.0) must be remediated or covered by complicated compensating controls as the only way out of remediating them.
• The traditional risk reduction driver has been around for much longer than compliance mandates. The first sale of Internet Security Scanner (ISS) 1.0 was reported in 1993. Security Administrator Tool for Analyzing Networks (SATAN), the first free network vulnerability assessment tool also came out in 1995 — 10 years before PCI DSS, 10 years before the HIPAA security rule deadline, and seven years before The Sarbanes-Oxley Act. The risk reduction driver compels organizations to "discover and plug the holes before hackers do." One of the key challenges is determining which holes to fix, and in what order, for maximum effect.

• Other examples of VA/SCA tool operation driven by the need to reduce risk include patch auditing (sometimes also prescribed by compliance) and rogue asset discovery. Today, this driver also presents itself as a need to tie VA/SCA data to security intelligence, overall situational awareness and other monitoring needs.

• The IT management driver is less prominent for VA/SCA tools given that they are primarily security tools. Still, using network assessment to discover assets can grow more important in this era of bring your own device (BYOD) and even bring your own computer (BYOC) because even unmanaged assets often need to be discovered and controlled.

• It is also important to note that IT management requirements drive system configuration management tools, such as patch management, much more than vulnerability assessment tools.

Broader Market Segmentation

Although vulnerability management comprises both vulnerability assessment and configuration management/remediation, audit and management products have separated into different categories over the last few years, thus making it necessary to craft different market definitions for VA/SCA and system management (as discussed in the Broader Vulnerability Management Context section).

It is important to look at other tools used in the vulnerability management process — in essence, at markets adjacent to vulnerability assessment. The mind map in Figure 2 defines how some of the products in the broader space relate to vulnerability assessment tools and what role they play in vulnerability management processes.

Source: Gartner (March 2012)

Segmentation of VA/SCA and Close Adjacent Markets

The VA/SCA market is characterized by de facto segmentation of vendors serving large enterprises (which routinely scan millions of scanned Internet Protocol [IP] assets) and vendors serving smaller organizations (which scan smaller networks down to a few IP addresses). The sizes of scanned networks also drive other principal differences in product capabilities, from mature remote component management to user interfaces, that evolved to serve the complex needs of large organizations.

In fact, what seems to matter for having a large number of enterprise deployment successes is whether the tool evolved in such large environments. In other words, a vendor's multiyear experience with successful scanning of large environments seems to matter more than many other factors such as delivery form factor (software, appliance, or software as a service [SaaS]) or reporting strength. For example, a well-designed and battle-hardened remote component management of your chosen VA software tool will make distributed vulnerability assessment much less painful. Tools from less-experienced vendors might be able to scan tens of thousands of IP addresses, but still suffer from issues with scalability, data management, remote component management, and asset parameter definition that can only be improved after learning painful lessons from enterprise deployments.

One of the most important adjacent technologies that is sometimes predicted to merge with VA technology is DAST, which is covered in depth in "Market Profile: Dynamic Software Security Testing, Web Scanning 2011."

In brief, what separates VA tools from DAST tools is the DAST tools' ability to discover new vulnerabilities in commercial software of select types and to discover vulnerabilities in custom applications. (However, VA and DAST tools can neither discover vulnerabilities at the business-logic level nor find completely unknown types of code vulnerabilities.)

DAST tools are commonly run by application security teams, ideally before they are taken to production. In that, VA tools are operational security tools, while DAST tools lean more toward being application development security tools. Despite the convergence of some function, the information in Table 1 will likely remain true for the foreseeable future.

Source: Gartner (March 2012)

DAST is more likely to converge with static application security testing (SAST) than with VA; see "Market Profile: Dynamic Software Security Testing, Web Scanning 2011" and "Dynamic Software Security Testing: Web Application Scanning Technology Assessment" for more details.

Market Evolution

Vulnerability assessment and the markets for vulnerability management tools in general are evolving in parallel with the changing IT landscape. From the first SCA tools like COPS² (launched in 1989) and the first VA tools like Internet Security Scanner (ISS; launched in 1993), the tools evolved over the last decades from simple scripts and programs, only usable by expert users, to vast enterprise frameworks, usable by most security practitioners with basic network security knowledge and tool-specific skills. What hasn't changed is that, in order to prioritize what remediation efforts yield the highest risk reduction, the tool user must still be aware of business context.

An earlier version of this report mentioned a set of vendors related to vulnerability management processes. Their fate (see Table 2) serves as a great demonstration of how the market has evolved with separation of duties for security assessment and operations teams in recent years.

Source: Gartner (March 2012)

Few vendors that focus on the security assessment/audit audience have been acquired, but most of those focusing on management have been absorbed by general-purpose system management vendors since 2007. Even prior to 2007, only two assessment tools were in fact acquired: Foundstone by McAfee and Stat from Harris by Lumension (then named PatchLink).

One additional example is the complete disappearance of patch management as a stand-alone offering. Even Lumension, the sole independent vendor from the management side, focuses on broader "endpoint management and security" rather than patch management or security configuration assessment alone. Gartner now considers patch management more of a capability than a market; it is included as a part of server and desktop management suites.

Vulnerability Market Trends

The primary trend that has been mentioned in the document already is the nearly complete separation between tools that manage and tools that assess or audit. Essentially, a wall has been built between the management and assessment capabilities. Although this likely did not occur intentionally, it will probably end up improving security by enhancing separation of duties (SOD). As a result, each category can expand: VA vendors now assess more things and management vendors manage more things. Notably, Gartner predicted this trend in 2007.

Dealing With Huge Numbers of Vulnerabilities Is Still an Issue

A periodic scan of a 100,000 node network often yields from 1 million to as many as 10 million findings, including everything from critical issues to minor informational data points. Enterprises are having a hard time prioritizing which vulnerabilities to remediate because the tools don't provide enough data or flexibility for prioritization. Several approaches have emerged and are discussed in the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology," but this is still a big challenge for the market. This trend has affected the market by creating a sort of internal segmentation between tools that can effectively serve the needs of enterprises and those that focus on smaller organizations.

Organizations Are Seeing the Limitations of Network Vulnerability Scanners vs. Custom Application Vulnerabilities

Organizations should place equal or larger focus on application-level and Web application vulnerability scanning and remediation. Network scanning still happens more often than application scanning, but application threats seem to surpass the infrastructure threats today. Traditional vulnerability management vendors are building more Web application assessment features or even separate modules that compete with dedicated DAST tools.

However, these capabilities reportedly don't measure up to dedicated Web application scanning products, but the gap has been shrinking for a few years. It is very likely that this gap will never be fully closed because dedicated Web application scanners can use intrusive techniques that require extensive tuning (for example, fuzzing). Such techniques are required to discover previously unknown vulnerabilities and can be used only during the development and not the production phase of the tested applications.

Thus, it is very likely that both markets will exist for the foreseeable future. Some organizations use their VA tool's Web application assessment capability to scan their production and preproduction Web applications, using the VA tool as "DAST-lite." On the other hand, organization seeking to perform in-depth production stage analysis should keep their DAST tools.

Compliance Remains a Strong Driver for Vulnerability Assessment as Well as Remediation

Although PCI DSS-driven external scanning has spread down market and reached true commodity status, other regulations (apart from possibly Federal Information Security Management Act [FISMA] and the United States Government Configuration Baseline [USGCB]) have not lead to dramatic expansion of vulnerability-scanning volume across enterprises. Thus, organizational risk assessment needs drive vulnerability scanning as much as or more than any single regulation, apart from PCI DSS. In some cases, security and risk reduction requirements lead to the adoption of VA (but not SCA) capabilities, although compliance demands lead to SCA tools — which results in organizations owning dedicated VA and SCA tools that do not integrate.

In addition to mandatory compliance frameworks, new standards are emerging and expanding in this area and are discussed in the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology."

Most VA Vendors Have Integrated Configuration Assessment Features Into Authenticated Scanning

This integration ranges from basic password policy checks to advanced application-level control analysis. This trend led to the current VA/SCA dual functionality. The use of dissolvable agents deployed during an authenticated scan allowed some vendors to achieve even deeper scanning across the tested systems. By now, most vendors have built the tools to assess federal networks using Security Content Automation Protocol (SCAP) and, specifically, to assess such networks versus USGCB (formerly known as FDCC) as well as to report the results in XML format for compliance analysis across agencies. It appears that more vendors are making an effort to integrate the capabilities closer together over time.

Despite such unity, many organizations still separate vulnerability scanning and configuration weakness assessment. Gartner research indicates that it is not uncommon for a customer to purchase a tool that does unified vulnerability and configuration scanning and then only use it for vulnerability assessment (sometimes even without credentials) — thus wasting resources. Table 3 illustrates such separation.

Source: Gartner (March 2012)

As a result, although some enterprises have great success unifying the assessment of systems for traditional known vulnerabilities, in many other cases, different people, tools and processes are in place for traditional vulnerabilities and problematic system configurations. Two trends seem to be operating in parallel — separation of duty in the VA market and operationalization of security in the system management market. A unified approach in the later case seeks to treat all vulnerabilities and configuration weaknesses in the same manner (despite the differences noted above) and seeks to resolve all such issues, whether the goal is risk reduction or compliance automation.

Adoption of Cloud-Based Management Services for Vulnerability Scanning Is Expanding

Adoption of cloud-based management services for vulnerability scanning is expanding due to simplified management of remote components, aggregated reporting and other management features. It should be noted that cloud management of local scanners is not the same as true scan platform multitenancy and can be applied to single-tenant approaches and individual scanner appliances (that store vulnerability data locally), thereby achieving advantages such as simplified scan appliance management and aggregated report reviews. A solution can be local and multitenant or Web-managed and single-tenant. Pioneered by Qualys more than 10 years ago, cloud-based VA/SCA management infrastructure hosting appears to be gaining in popularity. In fact, Gartner now predicts that by 2015, more than 25% of security devices and security agent software will be managed using SaaS models.

Also, the range of form factors for delivery of vulnerability assessment has expanded in the market. Traditional software and appliances have expanded to virtual machines and SaaS services (and SaaS managed appliances) as well as agents, transient agents and even browser plug-ins (that turn any machine on your network into a scanner). Even traditional software scanner vendors offer options for external scanning using shared, hosted or even multitenant SaaS scanner instances.

Zero-Day Scanning and Exploitation Tools Are Being Integrated

Finally, as a minor trend, so-called zero-day scanning using private research or privately shared vulnerability research (typically via commercial zero-day "clubs" such as Zero Day Initiative [ZDI], VeriSign iDefense and others) has been incorporated by a few vendors and offered to customers who have purchased access to such vulnerability information as well as a scanner that integrates with it. Admittedly, if vulnerability data is shared with the security community, it cannot be called zero-day anymore, but such privately shared vulnerability data might not become public for months, so it is effectively zero-day for the rest of world.

Along the same line, some vendors have embarked on tighter integration with exploitation tools (sometimes labeled penetration testing tools, although they only help automate a small portion of a professional penetration test). It is hard to say whether this presents an overall trend because customers have continuously chosen safe scanning over riskier exploitation methods since the early 2000s. Vendor integration of exploitation tools is further confirmed by the fact that most organizations choose to outsource their penetration testing (while keeping vulnerability assessment in-house); thus, exploitation tools would likely be used by third parties.

Market Disruptors

Beyond the immediate future (2012 to 2013), market disruptors have the potential to significantly change vulnerability assessment tools and vulnerability management practices.

A few market disruptors are coming our way. New asset types to be scanned, increased volume of vulnerability scanning (in part due to new assets such as virtual machines that are much easier to provision than physical servers), new dynamic device profiling methods, cloud computing's extension of enterprise borders, consumerization and unmanaged devices will all significantly disrupt the way enterprises perform vulnerability assessment and vulnerability management. In particular:

• New assets types can disrupt the market because virtualization, consumerization, mobile revolution and cloud computing may call for different approaches to security assessment and leave the existing VA tools behind. Select vendors are hard at work on capabilities for assessing such new targets.
• Cloud computing can disrupt the market because fuzzy lines of control create difficulties for performing traditional active scanning. Vendors will need to learn the cloud-specific access methods as well as shared environment practices to stay relevant.
• Large volumes of data can disrupt the use of today's tools due to increased sizes of scanned environments. Enterprise-ready VA vendors should be prepared to field dedicated data analysis tools; some vendors already have.

These disruptors will lead to further market segmentation of vendors that have started solving the above challenges effectively from those vendors that have not encountered them yet.

VA/SCA Tool Use Cases

In order to better compare the tools, it is important to explicitly define the common use cases for vulnerability assessment tools today.

Based on customer and vendor discussions, the following key use cases are the most common. Most, if not all, vulnerability assessment tool operators will be using the tools for at least one and likely two of these three core use cases:

• Risk reduction or traditional "look for holes before hackers find them" (mostly VA, but also SCA capabilities)
• PCI DSS vulnerability scanning or external VA plus somewhat internal VA plus SCA
• Compliance assessment automation, including internal policy compliance assessment (SCA mostly, but also VA)

Other use cases are also common:

• Patch management audit, change management audit and other remediation audits
• Asset, application and service discovery and mapping; rogue device discovery; and asset identification
• Data gathering for penetration testing
• Data gathering for metrics and key performance indicators (KPIs), risk modeling, and governance, risk and compliance (GRC) data input
• Data gathering for security monitoring

Less common, but also observed in the field, are the following use cases:

• Firewall access control list (ACL) audit
• Predeployment app testing or "poor man's" DAST

These use cases are discussed in depth in the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology."

Gartner research indicates that, despite a multitude of VA market participants, only a small group of vendors owns the enterprise segment of a broader VA market. These vendors compete mostly with each other and are present on most enterprise VA tool shortlists.

This comparative review focuses on the following key vendors, listed here alphabetically:

• eEye Digital Security
• McAfee
• nCircle
• Qualys
• Rapid7
• Tenable Network Security

Framework

The selection criteria for this analysis include:

• "Positive" or "Strong Positive" in "MarketScope for Vulnerability Assessment"
• Visible in enterprise competitive evaluations
• Having an extensive set of large enterprise customers
• Offering active scanning
• Using their own scanning engines, which are supported by their own vulnerability research
• Having notable VA and SCA capabilities
• Proven ability to support customers, even in very large environments
• Actively pursuing emerging areas such as virtualization

The vendors included in this analysis all have several years (some have more than a decade) of experience working with large enterprises on their vulnerability assessment programs; this experience seems to be the most important indicator for predicting success in such environments. Other interesting vendors that don't fit all the criteria but offer notable technologies are mentioned in the Alternative Approaches section.

A few notes apply to most VA/SCA tools discussed here in detail as well as some of those omitted. Most current VA vendors offer flexible deployment options: hardware appliances, virtual appliances and software. All VA vendors in this report also offer options for internal scanning as well as external scanning via SaaS or hosted facilities. In fact, even vendors who rely on traditional deployable software offer externally hosted scanning options.

Although vulnerability assessment tools have existed for more than 15 years, selecting the one that will work for your particular environment remains a challenge. In this comparison assessment, we focus on vendors that are often visible in enterprise evaluations. It is also interesting to note that when asked "who do you compete with in most cases?" the vendors in this group only name other vendors from the same group. Clearly, there's a core group of vendors that serves a majority of enterprise customers. It is very likely that, despite including other vendors in its evaluation, a large organization will end up with one of the vendors listed above.

This framework compares the vendors based on:

• Approach and core capabilities
• Use cases supported
• New environment assessment capabilities
• Prioritization approach
• Individual vendor comments

Approach and Core Capabilities

The VA/SCAN space constitutes a mature market in terms of core capabilities for traditional environments. All the vendors support almost all the core capabilities described in Table 4 in The Details section. They also all are proven to be able to support very large environments, if necessary.

However, vendors differ in a few ways across the core capabilities, including the use of agents. Specifically, one vendor can scan using installable agents in addition to active scanning, some use transient agents and others perform authenticated scanning only. Solutions also differ in the way they implement core capabilities as well as in their management and administrative technologies. Still, the main capability of VA tools is to identify vulnerabilities on the systems, and all scanners are expected to detect publicly revealed vulnerabilities soon after the information is out in the open.

Similarly, all vendors claim some degree of Web application scanning (DAST), but they differ in their approaches to it: Some fully integrate it into VA, some offer it as a free add-on and others offer separate DAST offerings.

Along the same lines, although the traditional VA capabilities have been converging across the tools for a few years, deep SCA capabilities still differ significantly in both depth (covering many configuration checks across the operating system and application and possessing especially detailed application knowledge) and breadth (covering as many platform as possible, down to mainframes and mobile devices).

It is important to note that vendors differ philosophically on their approaches to unifying or separating VA and SCA assessments. Some offer separate tools (only integrated at the reporting stage), while others treat vulnerabilities and configuration weaknesses in the same way and fully integrate the assessment.

Table 4 indicates vendor approaches to integrating security configuration assessment capabilities with their core vulnerability assessment capabilities.

Source: Gartner (March 2012)

Use Cases Supported

It is also important to be mindful of your primary use cases when selecting a VA/SCA tool:

• Risk reduction
• PCI DSS scanning
• Policy automation

Although all of the vendors discussed here address the three core use cases well, they differ in their approaches to combining the use cases. For example, some offer unified vulnerability assessment and security configuration assessment, while others suggest separate tools and separate practices. It is less common to separate PCI DSS external scanning (using PCI ASV offerings), but some vendors will offer integrated external and internal PCI DSS scanning, while others separate the two.

Table 5 illustrates tool support for core use cases.

Source: Gartner (March 2012)

New Environment Assessment Capabilities

Table 6 summarizes vendor approaches to assessment of new environments: cloud, mobile and virtual. Unlike the core capabilities, VA/SCA tools differ significantly in this area, and capabilities evolve very rapidly. Thus, these features need to always be tested carefully before deployment.

Source: Gartner (March 2012)

A majority of VA/SCA vendors are in the early stages of implementing their virtualization-specific features. At the same time, vendors report that 50% or more of scanned environments include virtual assets, so the urgency to implement virtualization support is there.

Some vendors are clearly ahead in implementing coverage for new environments and, what is even more important, in having those features "battle tested" by customers. Rapid7 reports a large percentage of customers using virtualization and is stronger in covering virtual platforms (based on the use of hypervisor APIs and vCenter discovery modes), while eEye seems to run more advanced coverage of mobile devices (based on current BES integration) as well as virtualization (based on VMware ThinApp integration) and cloud (based on Amazon Elastic Compute Cloud [EC2] integration). Tenable offers a unique approach to mobile devices via its passive assessment tool, Passive Vulnerability Scanner (PVS).

As with virtualization, a majority of VA/SCA vendors are in the early stages of researching and prototyping their cloud-specific features. Enterprise adoption of public cloud assets likely will tolerate such a slow start.

It is important to note that, even if your organization has not yet deployed virtualization and public cloud assets, it is still useful to compare the tools based on these areas because it is very likely that such technology deployments will happen in the near future. The approach to virtualization is the most tactical of the three because it is an area that will become a challenge for vulnerability and configuration assessment at literally all organizations in the near future.

Prioritization Approach

All listed vendors, with the exception of Tenable, offer proprietary, sometimes customer-customizable, vulnerability prioritization algorithms that go beyond high/medium/low proprietary and base CVSS community scores. Rapid7, eEye, McAfee and nCircle offer particularly advanced algorithms that incorporate many different dimensions of technical and business context in order to help the operator make a decision about remediation of vulnerabilities. The detailed discussion of all prioritization approaches is provided in the upcoming "Vulnerability Management Practices and Vulnerability Assessment Technology."

Despite the algorithm strengths, intelligently prioritizing the remediation of vulnerabilities without knowing the business context (as well as the opinion of the system owner, in many cases) is next to impossible. No algorithm provided by the vendor will prioritize, out of the box, which systems must never have open vulnerabilities exposed to attackers and which should be protected by multiple layers of safeguards and are unlikely to be exposed to threats.

Organizations that need to incorporate all such information and increase their degree of automation will need to look beyond their VA/SCA tools at dedicated tools for network-aware firewall management (for example, Skybox Security, RedSeal Networks and others) that can help solve the problem using the network topology information gleaned from routers, firewalls and other network devices. However, the organizations need to be prepared to budget roughly the same amount for these tools as they previously spent on vulnerability assessment tools (although the tools also have other capabilities beyond helping with vulnerability prioritization using their own scoring algorithms). See "Tools for Network-Aware Firewall Policy Assessment and Operational Support" for details.

Another option is a dedicated tool for managing vulnerability information. One such tool, Risk I/O, is produced by HoneyApps. It can integrate the data from multiple scanners (VA, DAST and even SAST), manage and analyze the scan results (using the customizable system of tags), present custom asset and vulnerability priorities, and run the remediation workflow process, thus tying multiple systems together. It should be noted that Risk I/O is designed to be used with DAST tools (or the DAST capability of a VA tool) that run scans on production systems and then track and measure the remediation activities.

Individual Vendor Comments

eEye Digital Security was less visible in competitive evaluations in recent years, but the company has definitely come back to the market with new VA/SCA technologies that focus on the areas of high interest to enterprises. Additionally, its long history with large deployments should be a value to enterprises looking to improve their vulnerability management practices.

McAfee Vulnerability Manager (MVM) remains a strong contender in the largest segment of the market, and its product team promises rapid development of more-advanced features that will focus solely on large enterprises. Its long history with large deployments should be a value to enterprises looking to improve their vulnerability management practices.

nCircle has a long history with complex enterprise deployments, which allows it to be of value to enterprises looking to improve their vulnerability management practices. nCircle works well when there is no need for integrated VA and SCA capabilities because these are provided via different tools.

Qualys is very visible in today's competitive evaluations. Additionally, its long history with large deployments should be a value to enterprises looking to improve their vulnerability management practices. Qualys works well when there is no need for tightly integrated VA and SCA capabilities, as long as customers are comfortable with the cloud delivery of VA/SCA scanning.

Although Rapid7 is a relative newcomer to enterprise vulnerability management, the company reports making major inroads toward addressing the needs of large customers in recent years. Its tool offers unified VA and SCA capabilities with flexible delivery models and leverages its Metasploit tool for vulnerability validation and in-depth testing.

Tenable Network Security's long history as a free tool and its recent enterprise and government deployment successes make Nessus a popular choice for organizations. Tenable Nessus combined with PVS also enables organizations to achieve vulnerability monitoring in challenging environments.

See The Details section for full vendor profiles.

Alternative Approaches

This section highlights some alternative approaches taken by companies in the market. Despite the separation between assessment and management tools, some organizations choose to use their IT management tools and older agent-based tools. System management tools such as IBM BigFix are occasionally used for assessment-only work as well. In general, agent-based tools experienced a decline but are re-emerging to support cloud environments (which cannot be scanned from the outside) and virtual machines (which are not always on).

Pure agent-based tools and IT management tools with security assessment capabilities are still around as well. Symantec Enterprise Security Manager (ESM), the most common agent-based assessment tool, still has sizable deployments, despite the fact that it is not being actively developed.

In Secunia's alternative, agent-based approach, an agent possesses extensive application discovery capabilities and can deduce vulnerabilities without any active scanning or probing the software. Unlike management agents, it is lightweight and is reportedly deployed in large environments.

Recommendations in this assessment focus on the VA market and vendor selection, as well as on comparison approaches.

For Large Deployments, Choose Vendors That Can Demonstrate Proven Experience With Similar Large Projects

The most reliable predictor of success for a large deployment is proven vendor experience with similar-scale enterprise deployments. Request vendor references for large deployments (i.e., similar in size to your organization) that have been running the tool under consideration for your project for at least two years. This time period is sufficient to reveal weaknesses with architecture and assessment approaches as well as the readiness of the vendor's organization to support larger enterprise projects.

Plan Your VA Deployment Architecturally, Then Talk to References

A large VA deployment project will encounter various issues at all phases, from planning to architecting to deployment to operation. In order to prepare for the most challenging problems, your organization should plan a VA deployment architecturally and then talk to references about their approaches and their views on your proposed approach. During such discussions, it is advisable to discuss multiple dimensions of product scalability: distributed assets, IP counts, reporting, workflow and scalability.

Compare Vendors on Core Capabilities First, but Don't Stop There

In order to select the best vendor to match your requirements, it is essential to compare vendors on core capabilities as well as other factors. Also compare vendors' approaches to emerging virtual, cloud and mobile environments. These environments are very likely to be common, and not all traditional vulnerability assessment and security configuration assessment approaches will work effectively in them.

Management is still the weak link for many product lines, but VA scan quality is now a commodity. SCA breadth and depth still differ across the tools. Evaluate products not only on the speed/accuracy of VA scans, but also on management/administration features and SCA coverage and depth (if SCA is required).

For Remotely Deployed Components, Carefully Verify the Extent of Centralized Management

Every enterprise-ready VA tool vendor promises fully centralized management of all components, from databases to remote scan engines, but the capabilities of tools vary widely. Tools on the market offer a range from "everything above hardware can be remotely managed" (often characteristic of SaaS tools) to minimum remote management. In the latter case, only some scan engine features can be centrally managed; the rest of the functionality can be remotely managed via SSH access to a scanner system. Obviously, such an approach will not scale to hundreds of scan engines deployed in the field.

If ease of management of the highly distributed VA tool is a key priority, then consider end-to-end (application and OS) cloud/hosted management tools that typically have easier deployment and operation. Such tools will allow security managers to focus their efforts on assessment and remediation and not on tuning the tools for optimal performance or running VA tool updates.

Use a Dedicated Product for Vulnerability Assessment

An organization should use a dedicated tool for VA/SCA. Specifically, do not use your network or system management product to perform vulnerability assessment in most situations (a rare exception is when an agent-based management product supports the required platform but the assessment product does not). Only VA tools using active network scanning can discover most types of network assets and assess the broadest range of assets for vulnerabilities. VA vendors have also expanded their SCA capabilities to rival those of legacy agent-based products.

Note that further expansion of consumerization will dramatically increase the number of unmanaged devices on most networks, thus increasing the need for a dedicated security assessment tool that can detect and assess them. The same trend will also further decrease the effectiveness of relying on server and PC management tools for security assessment.

Match the Vendor's VA/SCA Approach to Your Own

Select a vendor that uses the approach to VA and SCA that your organization already utilizes in its IT environment. If the same team is tasked with looking for vulnerabilities — generally weak configurations and technology policy validation — then pick a tool that follows the unified approach. If a security team is only tasked with VA scanning while system administrators and operators are tasked with finding and securing the configurations, then pick a tool that excels at VA.

Specifically, test both VA and SCA capabilities on operating systems and applications in your environment in order to check whether the depth of the resulting assessments is adequate. For example, does the tool go far enough with assessing configuration of Oracle databases? If not, invest in dedicated tools for these environments.

Know Where the General Purpose VA Tool Reaches Its Limits

Determine whether you need capabilities in more specifically focused tools such as DAST for application scanning or Database Audit and Protection (DAP) for database scanning. Specifically, know where your chosen VA tool Web application scanning module or database scanning module reaches its limits. General-purpose VA tools can be used to assess most environments, but when extra assessment depth is required, the organization should consider investing in dedicated tools. Custom applications, Web applications, databases and mainframes are all examples of situations in which special tools will have extra depth over general-purpose VA tools.

Investigate Vulnerability Assessment for New Environments Now

Check how the vendor assesses vulnerabilities and configurations for new environments, such as cloud, virtual and mobile. Multiple approaches can be used, and it is likely that only a combined approach can be used to assess the vulnerabilities across these rapidly evolving areas, which have few standards or even agreed-upon methods for performing security assessments. Test whether your organization can adopt the approach your vendor pursues, and ask for road maps with additional details on how the vendor plans to achieve both depth and breadth of coverage. Start scanning virtual environments for VA/SCA.

Steer Compliance-Focused Vulnerability Management Projects Toward a Risk-Based Approach

Expand the compliance scanning into risk reduction and (where applicable) technical policy assessment. Enterprise risk reduction is a better driver for enterprise vulnerability management projects in regard to delivering value for both compliance readiness and improving security. In particular, expand external vulnerability scanning beyond PCI DSS compliance into an external and internal vulnerability management program that's focused on reducing risk to IT assets.

This section includes additional details on vendors and the market.

Vendor Profiles

The following section covers the profiles on the selected key vendors, which are presented alphabetically.

eEye Digital Security

eEye Digital Security, founded in 1998, offers the eEye Retina family of vulnerability assessment products and the Blink Endpoint Protection product (aka Retina Protection Agent). Specifically, the eEye products relevant to this report are:

• eEye Retina CS system, including scanner management, reporting and a dedicated data analysis component (Retina Insight)
• eEye Retina Network Security Scanner with VA and SCA capabilities
• eEye Blink HIPS product, which can now be used for agent-based scanning as well (Windows platform only)

eEye does not offer a separate Web application scanning tool, but includes minimal (compared with dedicated DAST tools) Web application scanning capabilities in the main VA product. All products have been developed by eEye Digital Security itself, which also maintains a vulnerability research and consulting unit that includes a penetration-testing service. eEye products are offered as software, appliances or virtual images.

According to eEye, its Retina products have been used in extremely large environments for the last 10 years across both government and commercial verticals. The vendor reported having more than 9,000 customers for its VA solution and indicated that some of its customers routinely scan networks of more than 1 million assets.

Retina CS closely integrates VA and SCA capabilities for scanning, data management and reporting. No additional product or product modules are required to assess configuration using either eEye native or SCAP-compliant Open Vulnerability and Assessment Language (OVAL) methods (used for FDCC/USGCB U.S. federal government environment scanning).

Large Retina environments are architected using one (rarely: more) Retina CS management console and multiple Retina Network scanners. Blink Windows agents are sometimes used for vulnerability assessment where active network scanning is impossible. Maximum recommended asset count is 10,000 per scanner instance. For distributed deployments, the vendor recommends placing scanners closer to assessed assets to avoid scanning through low-bandwidth links and firewalls.

Retina vulnerability assessment is being aggressively expanded into new environments such as virtual, cloud and mobile. eEye offers the ability to scan mobile devices (for example, BlackBerrys via BES) and assess virtual assets for vulnerabilities and configuration weaknesses. VMware ThinApp software development kit (SDK) is utilized for assessing the security of virtualized applications, which is unique to eEye. A connector to VMware vCenter is planned in the near future, and CIS configuration templates for VMware ESX are currently in production use. On the cloud side, native connectivity (via provider APIs) is also planned for the near future.

Retina offers a vulnerability prioritization algorithm that includes CVSS base, temporal and, if configured by the customer, environmental scores. Vulnerabilities are cross-linked to available exploits, and malware and can be correlated with threat information, which is collected from Blink and third-party products. Asset values can be defined using SmartGroups to avoid individual asset value assignment. Additional visualization tools (HeatMaps) provide a way to sort through the scoring data and determine the order in which vulnerabilities have to be fixed. Exploit availability information comes from third-party integrations such as MetaSploit and Core.

eEye Retina CS can be used for both the traditional vulnerability scanning and the security configuration policy scanning core use cases, and it shines where these types of assessments can be integrated inside the organization. PCI DSS scanning is also available, and eEye is listed as a PCI ASV. eEye's Patch Management Module also enables the organization to better leverage Windows Server Update Services (WSUS) for remediation activities.

Additional interesting capabilities of eEye VA/SCA technologies include agent-based scanning through Blink, expanding mobile device scanning, integration of threat data from Blink and other endpoint protection solutions, and a dedicated analytics console (Retina Insight), which is useful for large-scale vulnerability data management.

McAfee

McAfee, founded in 1989, offers the McAfee Vulnerability Manager (MVM) vulnerability assessment product, as well as a broad range of other security products. Specifically, some of their products relevant to this report are:

• MVM is McAfee's main VA product.
• McAfee PCI Certification Service (McAfee Secure) offers dedicated PCI DSS scanning and is focused on smaller organizations.
• McAfee Policy Auditor is an agent-based configuration assessment tool that uses SCAP.
• McAfee ePolicy Orchestrator (ePO) is McAfee's endpoint security management layer.
• McAfee Risk Advisor helps analyze the data coming from other McAfee products such as MVM and ePO.

McAfee does not offer a separate Web application scanning tool, but includes minimal (compared with dedicated DAST tools) Web application scanning capabilities in the main VA product. McAfee acquired MVM from Foundstone in 2004, McAfee PCI Certification Service from ScanAlert in 2007 and McAfee Policy Auditor from Citadel Security in 2006. McAfee developed ePO and Risk Advisor itself. McAfee also offers dozens of host, database and network security products and maintains vulnerability and threat research as well as extensive consulting units and a PCI DSS Qualified Security Assessor (QSA) unit.

According to McAfee, MVM has been used in extremely large environments in both government and commercial verticals, with some customers having up to 4 million scanned IP assets. McAfee reported thousands of customers for its VA solution.

MVM closely integrates VA and SCA capabilities for scanning, data management and reporting. No additional products or product modules are required to assess configurations. For environments that prefer agents, McAfee Policy Auditor can also be deployed for SCA scanning using SCAP OVAL content. MVM also allows agentless OVAL scanning, if required.

Large MVM environments are architected using a traditional n-tier model, including a central database, application server (for reporting and management) and a number of scanning engines. A very large organization may also choose to deploy multiple instances of the above and then tie them together for central and regional data analysis. At about 10,000 scanned assets, the McAfee recommends separating components to a couple of servers and using multiple scanning engines, especially if the environment is globally distributed. The largest environments commonly use clustered SQL servers and storage area network (SAN) arrays and span three to five MVM servers and hundreds of scan engines in the field, all reporting to the same MVM multiserver instance. Isolated or very bandwidth-constrained remote networks can use a separate MVM instance that is later synchronized with the master MVM. McAfee reports that MVM is successfully used aboard Navy ships and cruise ships with limited and intermittent connections.

McAfee is working on expanding MVM coverage into new environments such as virtual, cloud and mobile assets as well as non-PC network devices (for example, Xerox printers or APC power backup controllers). MVM can discover mobile computing assets such as Apple iPad or iPhones on Wi-Fi networks and can detect vulnerabilities and some configuration weaknesses on them. Also, MVM can deduce the mobile asset security information from the iTunes repositories found on Windows/Mac systems. MVM can assess virtual platforms (VMware ESX/ESXi) via traditional SSH remote access, and native API-based assessment is in the works. For infrastructure as a service (IaaS) scanning, customers can deploy one or more scan engines inside the IaaS environment in VM form and control that engine from their existing on-premises MVM instance. In the future, McAfee plans to expand VA/SCA scanning using ePO agents to make it more usable in cloud environments.

McAfee MVM offers a vulnerability prioritization algorithm; McAfee Risk Score is based on a vulnerability's impact, severity and exploit availability and popularity. In addition, standard National Vulnerability Database (NVD) CVSS base and McAfee CVSS scores (base and temporal) are available. Another algorithm, FoundScore, incorporates asset criticality, risk of discovered vulnerabilities, scanned resource type and so on. This tunable algorithm incorporates some configuration information (for example, visible network services and essential services) in its scoring. Asset value can be assigned using an extensive system of rules and tags. Its ultimate goal is to automate asset discovery, vulnerability assessment and even remediation prioritization.

MVM can be used for both the traditional vulnerability scanning and the security configuration policy scanning core use cases, and it shines where these types of assessments can be integrated inside the organization. PCI DSS scanning is also available from McAfee, but McAfee Secure, not MVM, is the product listed as a PCI ASV tool. Thus MVM should be used for internal PCI DSS scanning only. MVM is reported to be used by several organizations to attain PCI ASV status.

nCircle

nCircle, founded in 1998, offers a family of vulnerability assessment products as part of a larger suite. The nCircle products relevant to this report are:

• nCircle IP360 vulnerability scanner, offering VA capabilities (delivered via Device Profiler scanning engines), VA management (via VnE Manager appliances) and an integrated Web application scanner (nCircle WebApp360)
• nCircle Configuration Compliance Manager with SCA capabilities and its own management console and integrated file integrity checking capabilities
• nCircle Suite360 Intelligence Hub management console for integrated reporting over VA and SCA as well as data analysis
• nCircle Benchmark for organization-specific and shared metrics and benchmarks, including vulnerability assessment products

nCircle also offers a dedicated Web application scanning tool: WebApp360. nCircle developed IP360 and acquired Configuration Compliance Manager Cambia Security in 2007. nCircle continues to develop the products separately, but their data (but not management) is integrated via Suite360 Intelligence Hub. The products are offered as appliances, software or virtual images. The company also operates a vulnerability research team and offers professional services.

According to nCircle, its products have been used in large environments in both government and commercial verticals. nCircle reported having more than 5,000 customers for its VA/SCA solutions.

nCircle requires separately priced products and separate scans (but uses the same physical appliance — Device Profiler — to utilize VA or SCA capabilities) and does not closely integrate VA and SCA for scanning and management. Reporting is integrated, but via a separately priced product. No additional products or product modules are required to assess assets using SCAP-compliant OVAL assessment methods (both IP360 and CCM are SCAP-validated).

Large nCircle environments use a single management appliance (VnE Manager) and several scanning appliances (Device Profilers). Maximum recommended asset count, assuming daily scans, is 5,000 assets per Device Profiler. For distributed deployments, nCircle recommends placing Device Profilers closer to assessed assets to avoid scanning through low-bandwidth links and firewalls. For environments performing both VA and SCA scanning, an additional CCM management console is required (but no new scan engines), and nCircle Suite360 Intelligence Hub is strongly recommended for integrated reporting and data analysis.

nCircle CCM can perform configuration assessment of hypervisors via a certified CIS policy for VMware; additional custom checks can be created.

nCircle offers a vulnerability prioritization algorithm, nCircle Risk Score, that incorporates vulnerability severity, exploit availability and duration of time that the vulnerability has been reported public. System asset values can also be used to improve remediation prioritization accuracy. As with all such products, customers must assign their asset values to get full benefits. nCircle CCM has its own scoring algorithm that focuses on the risk of detected configuration changes that is not tied into the main Risk Score.

nCircle can be used for both the traditional vulnerability scanning and security configuration policy scanning core use cases. Note that using the solution for both VA and SCA will incur additional costs and management complexity. PCI DSS scanning is also available, and nCircle is listed as a PCI ASV.

Additional interesting capabilities of nCircle include its unique delivery model — PureCloud — that can deploy a transient agent via a browser and turn any system into a vulnerability scanner for internal vulnerability scanning (currently VA only, no SCA).

Qualys

Qualys, founded in 1999, offers the QualysGuard suite of IT security and compliance assessment services. QualysGuard services are offered directly to customers and through consultants, resellers and managed security service providers.

The Qualys products relevant to this report are:

• QualysGuard Vulnerability Management (VM) is the main vulnerability assessment service.
• QualysGuard Policy Compliance offers agentless security configuration assessment.
• QualysGuard PCI Compliance offers dedicated PCI DSS external scanning.

QualysGuard VM and PC can be used for both external and internal scanning by means of SaaS appliances that are available as either physical hardware or virtual images. An appliance can be installed on the customer's network for internal vulnerability scanning while external scans are conducted across the Internet from one of the Qualys' Secure Operations Centers (SOCs). QualysGuard supports multiple appliances for each user account, and the appliances can be deployed as necessary depending on the architecture of a customer's network. All appliance scanning and Web scanning is controlled from the same Web interface. Appliances are fully managed from the cloud (no local management); all scan data is stored in the cloud.

Qualys offers a separate Web application scanning service — QualysGuard Web Application Scanning (WAS) — and includes minimal Web application scanning capabilities in the main VA product. All products have been developed by Qualys. Unlike other vendors, Qualys does not operate any consulting units and relies on partners for services. Qualys products are offered as SaaS only; however, in special cases, the vendor may agree to have the entire SaaS platform hosted on the premises of the customer (aka the @customer option).

According to Qualys, its QualysGuard products have been used in extremely large corporate environments. It reported having more than 5,500 customers, including 50 of the Fortune 100, for its VA solutions, and indicated that some of its customers routinely scan networks of more than 1 million assets.

Qualys does not currently integrate its VA and SCA capabilities for scanning, data management and reporting, but they can be accessed from the same Web interface. QualysGuard VM focuses on VA (with some SCA capabilities for historical reasons) while QualysGuard PC focuses on SCA capabilities. The products are separately priced, require separate scans and result in separate reports, but they can be launched from the same appliances using the same asset groupings, authentication information and permissions structure. SCAP OVAL scanning for FDCC/USGCB compliance requires an add-on module (QualysGuard FDCC) for QualysGuard PC; it also requires the launch of special SCAP scan types. In light of this, Qualys recommends that customers VA scan all IP ranges but focus SCA scanning on key servers in data centers. Still, Qualys promises tighter integration between VA and SCA capabilities.

Large QualysGuard environments are architected using multiple QualysGuard appliances, which are always managed by a cloud platform. For distributed deployments, Qualys recommends placing scanners closer to assessed assets to avoid scanning through low-bandwidth links and firewalls. Multiple scanners may be used in parallel to improve performance or accommodate small scan windows. Every additional appliance carries a nominal cost, in addition to the cost of the scanning service.

Qualys pioneered cloud delivery of VA/SCA capabilities more than 10 years ago. Now it is planning to expand into more dedicated cloud asset scanning (via EC2-specific and virtual private cloud [VPC]-specific scan modes) and dedicated scanning of hypervisors (via vSphere and vCenter using the VMware APIs). Qualys can currently detect vulnerability and configuration weaknesses in many non-PC devices such as network equipment, printers and networked power backups.

As with many other vendors, an internal workflow is provided within QualysGuard, and external ticketing systems are supported via an API. With Qualys, such integration with external ticketing systems is bidirectional, so if the ticket status changes, the change will be reflected in QualysGuard. Tickets are only closed after a follow-up scan reports that the vulnerability is no longer present.

Qualys offers a vulnerability prioritization algorithm, Business Risk, that combines vulnerability severity with user-defined asset values. The algorithm also includes serious configuration issues such as unsecure Secure Sockets Layer (SSL) configuration that uses weak encryption, end-of-life or obsolete installed software (from those detected by QualysGuard VM — because QualysGuard PC data is no included in this computation). Qualys also allows users to filter vulnerabilities by CVSS base score, CVSS vector parameters, exploit availability (via integration with third-party tools such as Core Impact and Metasploit), remediation (availability of vendor patches, virtual patches or workarounds), affected OS or software application, and other attributes. The customer can further prioritize remediation activities based on the number of hosts affected, the age of the vulnerability and so on.

QualysGuard VM can be used for the traditional vulnerability scanning, PCI DSS scanning (internal and external because Qualys is an ASV) and security configuration policy scanning core use cases. Organizations focused only on PCI DSS external scanning can choose to utilize a dedicated PCI scanning platform that has a much lower price point. Qualys' SaaS model shines in large environments because the task of managing a large number of scan engines is fully automated by the cloud platform. Because Qualys stores and encrypts all vulnerability data in its data centers, customers who choose QualysGuard should be comfortable with that model.

Rapid7

Rapid7, founded in 2000, offers the Nexpose family of vulnerability assessment products and the Metasploit tools, which are useful during penetration testing and for vulnerability validation. The Rapid7 products relevant to this report are:

• Nexpose Enterprise is a unified VA and SCA solution.

Rapid7 includes a unified VA product with basic (compared with dedicated DAST tools) Web application scanning capabilities. Rapid7 developed the product itself, and it maintains consulting units that focus primarily on penetration-testing services using Metasploit tools. The products are offered as appliances, software, virtual images, a hosted service or a multitenant cloud service. In general, services play a key role in Rapid7 offerings, unlike some vendors that shy away from them. Rapid7 also offers professional services for deployment as well as dedicated security assessment services.

According to Rapid7, its Nexpose products have been used in large environments. It reported having more than 1,700 customers for its VA/SCA solution. Some of its customers routinely scan networks of tens of thousands of assets.

Rapid7 Nexpose closely integrates VA and SCA capabilities for scanning, data management and reporting. No additional products or product modules are required to assess configuration using either native or SCAP-compliant OVAL assessment methods (used for FDCC/USGCB U.S. federal government environment scanning).

Large Nexpose environments are architected using one Nexpose Security Console and multiple Nexpose Scan Engines. All scan results are stored on the central management Console system that is used for both reporting and management of scans. The same technology, with a native multitenant architecture, is used for Rapid7 SaaS scanning deployments. For distributed deployments, the vendor recommends placing scanners closer to assessed assets to avoid scanning through low-bandwidth links and firewalls. Still, the vendor reported that its technology was successfully used for scanning over slow 56 Kbps links.

Rapid7 is aggressively expanding vulnerability assessment into new environments such as virtualization. Rapid7 has developed vAsset Discovery technology that natively integrates with VMware vCenter using APIs to enable reliable discovery of virtual environments.

Nexpose offers one of the most advanced vulnerability prioritization algorithms of any of the tools in this space. In fact, it has multiple predefined prioritization mechanisms or custom algorithms to evaluate risk in an environment. Nexpose also includes exploit and malware availability information, detected running software and OS, CVSS vector information, vulnerability age, and user-defined asset parameters.

Rapid7 Nexpose can be used for both the traditional vulnerability scanning and the security configuration policy scanning core use cases, and it shines where these types of assessments can be integrated inside the organization. PCI DSS scanning is also available, and Rapid7 is listed as a PCI ASV. Any of the Metasploit editions (Community, Express or Pro) can be used together with Nexpose to improve penetration testing by internal security teams. Using Metasploit in conjunction with Nexpose actually allows a closed-loop system of scanning for vulnerabilities by Nexpose, validating with Metasploit which vulnerabilities are actually exploitable and then creating automated Nexpose exceptions to report which vulnerabilities can be ignored because mitigating controls are in place.

Additional interesting capabilities of Rapid7 include being able to use credentials discovered during a scan for authenticated scanning, remediation plan reports with prioritized actionable guidance and Nexpose flexible deployment options, including the hosted and shared SaaS models.

Tenable

Tenable Network Security, founded in 2002, offers the Nessus vulnerability assessment product, as well as Tenable PVS, a unique passive vulnerability scanner. The Tenable products relevant to this report are:

• Nessus vulnerability scanner (available from 1998 as an open source tool, but later closed) is a traditional active vulnerability scanner that combines both VA and SCA capabilities.
• Tenable Passive Vulnerability Scanner (PVS) finds vulnerable applications and platforms by sniffing network traffic.
• SecurityCenter provides a management console for these and other Tenable products (such as Log Correlation Engine)

Tenable does not offer a separate Web application scanning tool, but it includes minimal (compared with dedicated DAST tools) Web application scanning capabilities in the main VA product. All products have been developed by Tenable security personnel. The products are offered as software, appliances or virtual images as well as SaaS for external scanning (branded as Nessus Perimeter Service).

According to Tenable, its Nessus products have been used in extremely large environments and in both government and commercial verticals. Tenable reported more than 10,000 customers and more than 75,000 users for its VA solution (including the freeware version). Some of its customers routinely scan networks of more than 1 million assets.

Nessus closely integrates VA and SCA capabilities for scanning, data management and reporting. No additional products or product modules are required to assess configuration using either Nessus native or SCAP-compliant OVAL assessment methods for FDCC/USGCB scanning.

Large Nessus environments are architected using one SecurityCenter management console and multiple Nessus and PVS scanners. Multiple consoles can be deployed in a hierarchical manner for the largest deployments for regional and aggregated reporting and management. The maximum recommended asset count is 8,000 assets per scanner. For distributed deployments, the Tenable recommends placing scanners closer to assessed assets to avoid scanning through low-bandwidth links and firewalls. In situations where bandwidth is scarce or strict ACLs are in place, Tenable recommends authenticated scanning because it is more efficient and accurate.

Tenable vulnerability assessment is expanding into new environments such as virtual, cloud and mobile. Virtual systems are currently scanned for vulnerabilities and configuration weakness. CIS configuration policy for VMware ESX is also available among its SCA capabilities. VMware vCenter integration is also in Tenable's immediate plans. In mobile, Tenable leverages PVS to detect mobile systems on Wi-Fi networks and to detect vulnerable applications and OSs, such as outdated iPhone or iPad iOS versions. Tenable's answer to the cloud is also PVS: It can help sniff access to Amazon or other cloud resources.

Tenable does not offer a vulnerability prioritization algorithm. Instead, Tenable customers rely on an extended system of filters that help refine the scan results by Nessus plug-in ID, CVE ID, CVSS score, vulnerability age, exploit availability, and so on and event log information (for customers using the Tenable Log Correlation Engine [LCE] tool) and patch availability (via third-party system integration). No asset roles or values can be defined inside the product due to lack of customer interest in such capabilities; Tenable offers other asset management capabilities such as grouping (static and rule-based) and import.

Tenable Nessus, together with Security Center console, can be used for both the traditional vulnerability scanning and the security configuration policy scanning core use cases, and it shines where these types of assessments can be integrated inside the organization. Tenable is working toward an ASV certification, but can currently be used for internal PCI DSS scanning.

Additional interesting capabilities of Tenable's VA/SCA technologies include scanning for sensitive data files, including Social Security numbers (SSNs) and primary account numbers (PANs) using the Windows File Contents Compliance Checks plug-in; detecting sensitive data in transit via PVS; and even creating dynamic asset lists that can be used for reporting and scan targeting based on sensitive data discovery.

VA/SCA Core Capabilities

Gartner defines a high-level list of vulnerability assessment critical capabilities in "Evaluating Vulnerability Assessment Capabilities." For the purposes of this assessment, it is useful to expand this set of capabilities with additional granular capabilities as well as capabilities present only in enterprise-focused tools.

Table 7 shows the expanded capabilities mapped to core use cases.

Source: Gartner (March 2012)

These capabilities are present in all leading products on the market, as discussed in this assessment. However, note that products differ in their relative strengths and weaknesses.

Table 7 can also be used to compare products not profiled in this comparison assessment.

Continuing concerns about regulatory compliance (PCI DSS) are driving the vulnerability assessment market and the expansion of security configuration assessment capabilities. However, regulatory compliance is not the only driver: Risk reduction and internal policy compliance drive VA/SCA tools as well. Assessment-only products have separated themselves from management products to provide a means for security groups to report on operations activities. The VA market also segmented internally into a core group of vendors that owns most of the enterprise VA market share.

Vulnerability management is "the key process for finding and remediating security weaknesses before they are exploited." The parts of this process are "policy definition, assessment, shielding, mitigation and monitoring" (see "Understanding Vulnerability Management Life Cycle Functions").

Defined in the forthcoming "Vulnerability Management Practices and Vulnerability Assessment Technology."

² COPS is not an abbreviation, but a tool name. Originally it meant "Computerized Oracle and Password System."

³ McAfee Secure is the McAfee vulnerability assessment certified by the PCI Council as PCI ASV tool, not the MVM discussed in this comparison assessment.