LICENSED FOR DISTRIBUTION

Innovation Insight for Security Orchestration, Automation and Response

Published: 30 November 2017 ID: G00338719

Analyst(s):

Summary

Enterprises are striving to keep up with the current threat landscape with too many manual processes, while struggling with a lack of resources, skills and budgets. Security and risk management leaders should determine which SOAR tools improve security operations efficiency, quality and efficacy.

Overview

Key Findings

  • Security operations teams struggle to keep up with the deluge of security alerts from an increasing arsenal of threat detection technologies.

  • Security operations still primarily rely on manually created and maintained, document-based procedures for operations, which leads to issues such as longer analyst onboarding times, stale procedures, tribal knowledge and inconsistencies in executing operational functions.

  • The challenges from an increasingly hostile threat landscape, combined with a lack of people, expertise and budget are driving organizations toward security orchestration, automation and response (SOAR) technologies.

  • Threat intelligence management capabilities are starting to merge with orchestration, automation and response tools to provide a single operational tool for security operation teams.

Recommendations

IT security and risk management leaders responsible for security monitoring and operations should:

  • Assess how SOAR tools can improve the efficacy, efficiency and consistency of their security operations by using orchestration and automation of threat intelligence management, security event monitoring and incident response processes.

  • Focus on automating tasks and orchestrate incident response starting with procedures that are easy to implement and where machine-based automation will reduce incident investigation cycle times.

  • Use external threat intelligence as a key way to improve the efficacy of security technologies and processes within the security operations program.

Strategic Planning Assumption

By year-end 2020, 15% of organizations with a security team larger than five people will leverage SOAR tools for orchestration and automation reasons, up from less than 1% today.

Analysis

Security and risk management leaders responsible for security monitoring and operations face an increasingly challenging world. Attackers are improving their ability to bypass traditional blocking and prevention security technologies, and end users continue to fall victim to attackers through social engineering methods, while still failing to carry out basic security practices well. While mean time to detect threats may be trending down across industries, 1 it still takes way too long. Once detected, the ability to respond to, and remediate, those threats is still a challenge for most organizations. Additionally, many security teams have overinvested in a plethora of tools. As a result, they are also suffering from alert fatigue and multiple console complexity and facing the challenges in recruiting and retaining security operations analysts with the right set of skills and expertise to effectively use all those tools. This is all playing against a backdrop a growing attack surface that is no longer restricted to on-premises IT environments.

The attack surface today encompasses multiple forms of cloud (SaaS, IaaS and PaaS) and mobile environments, and even extends to third-party organizations that are suppliers to upstream organizations. Finally, effective security monitoring requires not only tools and well-documented incident response processes and procedures, but also the ability to execute them with consistency and precision, and the capability to refine and update responses as best practices emerge. Many organizations have few, if any, of these procedures documented. Sometimes they are just monolithic and inflexible, and continue to rely on ad hoc responses over and over again.

Since Gartner's first analysis of the SOAR space (which was initially defined by Gartner as "security operations, analytics and reporting"), the vendor and technology landscape has evolved. In 2017, many technologies claim the ability to orchestrate incident response, but present some limitations in capabilities that could deliver real overall benefits for the efficacy of an operations team. Examples of these shortcomings include a limited ability to show the big picture of organizations' state of security or the lack of connectivity to the organization's ecosystem of tools. Security orchestration and automation have become closely aligned with security incident response (SIR) and general operations processes. Security information and event management (SIEM) technology vendors have incorporated automated response capabilities to various levels of capabilities. Automated response is also appearing in other security technologies as a feature. The lack of centralized capabilities in the above solutions leaves security teams with a responsibility to manually collect and stitch together all this information, and work with manual playbooks for tasks related to each type of incident.

Figure 1 shows a continuous set of activities that can be performed by an SOC team by using SOAR technology. The figure reflects the use of the CARTA strategy for continuous monitoring and visibility.

Figure 1. SOAR Overview
Research image courtesy of Gartner, Inc.

Source: Gartner (November 2017)

Definition

Gartner defines security orchestration, automation and response, or SOAR, as technologies that enable organizations to collect security threats data and alerts from different sources, where incident analysis and triage can be performed leveraging a combination of human and machine power to help define, prioritize and drive standardized incident response activities according to a standard workflow. SOAR tools allow an organization to define incident analysis and response procedures (aka plays in a security operations playbook) in a digital workflow format, such that a range of machine-driven activities can be automated.

The Evolution of SOAR From 2015 to 2017

In 2015, Gartner described SOAR (then described as "security operations, analytics, and reporting") that utilized machine-readable and stateful security data to provide reporting, analysis and management capabilities to support operational security teams. Such tools would supplement decision-making logic and context to provide formalized workflows and enable informed remediation prioritization.

As this market matures, Gartner is witnessing a clear convergence among three previously relatively distinct, but small, technology markets (see Figure 2). These three are security orchestration and automation, security incident response platforms (SIRP), and threat intelligence platforms (TIP).

Figure 2. Convergence of SOAR Tools
Research image courtesy of Gartner, Inc.

SOA: security operations automation; TVM: threat and vulnerability management

Source: Gartner (November 2017)

The majority of solutions that Gartner tracks are mostly related to core security operations functions, such as responding to incidents, which are addressed by existing tooling (for example, a SIEM). SOAR integrates dispersed security data, and provides security teams with the broad functionality to respond to all types of threats. SOAR also enables processes that are more efficient, accurate, and allow for automation for common subtasks or an entire workflow. The primary target for a SOAR solution is the security operations center (SOC) manager and analysts responsible for incident response.

Gartner is also tracking an increasing role of SOAR functionality among TIP vendors. Indeed, SOAR's central role in the SOC makes it ideally suited to validate the quality of the threat intelligence used in an organization. By confirming alerts as true positive or false positive, SOARs can confirm or infirm the threat intelligence used to come to that conclusion. Likewise, the SOAR can now push validated threat intelligence to all the tools and security controls in the organizations that can take advantage of the indicators of compromise for local enforcement.

Description and Functional Components

SOAR can be described by the different functions and activities associated with its role within the SOC, and by its role with managing the life cycle of incident and security operations:

  • Orchestration — How different technologies (both security-specific and non-security-specific) are integrated to work together

  • Automation — How to make machines do task-oriented "human work"

  • Incident management and collaboration — End-to-end management of an incident by people

  • Dashboards and reporting — Visualizations and capabilities for collecting and reporting on metrics and other information

In the following sections, we will review each of these functions in more detail.

What SOAR is not:

  • Governance, risk and compliance (GRC), where the focus is on managing adherence to compliance frameworks, often based on controls. Gartner has evolved GRC to be called Integrated Risk Management (IRM) now to include both IT risk management and Audit and Risk management.

  • SIEM, which provides reliable log ingestion and storage at scale, as well as normalization and correlation of events for real-time monitoring and the automated detection of security incidents.

  • User and entity behavior analytics (UEBA) or advanced threat detection, which are focused on behavioral and network analysis or the detection of indicators of compromise.

  • Threat and vulnerability management, which provides awareness for the types of threats facing an organization. TVM is focused on identifying, prioritizing and remediating security weaknesses based on potential risk and impact of vulnerabilities.

Drivers for SOAR include:

  • Staff shortage: Due to staff shortages in security operations (see "Adapt Your Traditional Staffing Practices for Cybersecurity" ), there is a growing need to automate, streamline workflows and orchestrate security tasks. Also, the ability to be able to demonstrate to management the organization's ability to reduce the impact of inevitable incidents is ever-present.

  • The explosion of unattended alerts from other security solutions: The process of determining whether a specific alert deserves attention requires querying many data sources to triage.

  • Threats becoming more destructive: Threats destroying data, the disclosure of intellectual property and monetary extortion require a rapid, continuous response with fewer mistakes and fewer manual steps.

  • The need to better understand the intersection of your environment with the prevailing threat landscape: A large number of security controls on the market today benefit from threat intelligence. SOAR tools allow for the central collection, aggregation, deduplication, enrichment of existing data with threat intelligence, and, importantly, converting intelligence into action.

Orchestration

Gartner sees orchestration as the ability to coordinate informed decision making, and formalize and automate responsive actions based on measurement of the risk posture and the state of an environment. SOAR orchestrates the collection of alerts, assesses their criticality, coordinates incident response and remediation, and measures the whole process. One example is the response to reported email that may be suspicious. The end user reports to the SOC a suspicious email, which would require an investigation to confirm whether sender has a bad reputation (through threat intelligence). The use of DNS tools would confirm origin of the email. The analyst would have to extract any hyperlink from the email to validate through URL reputation, or to detonate the link through a secure environment or to run attachments on a sandbox. This process would be done for every reported suspicious email to transform it to an incident. Orchestration provides enough information (automating the data collection into a single place) to help the analyst to review and decide if the situation is suspicious. If the investigation confirms an incident, it would initiate the workflow (playbook) to respond to the incident. Integration with the email system, sandbox and ticket system would provide an automated process to look at the email system to find all messages with a suspicious link or attachment. Then, the system would quarantine email that was sent to other users, while waiting for the decision of deleting or allowing access to quarantined email. Think of the process as conducting an orchestra: a conductor controls multiple musical instruments to produce not just noise, but music. Today, security teams have the problem of having to pick up and play each instrument, but they can't play many instruments at the same time. It takes time to pick and up put down each instrument. In the world of security operations, this is called "context switching," and it costs teams time (dead time) to orchestrate and perform each step in a process.

Table 1 outlines the main requirements for orchestration in SOAR tools.

Table 1.   Summary of Orchestration Capabilities

Capability

Minimum Requirements

Basic integration

A wide range of out-of-the-box integration connectors to other security solutions. Today, the list of supported vendors might not cover all the technologies you have in your environment.

Bidirectional integration

Multiple action types can be described at a high level as "push" or "pull." "Push" means telling a tool/device to do something. "Pull" means connecting to a tool/device and requesting information it might have. Gartner recommends that end users press their tool vendors to support a full range of both push/pull type capabilities via a well-documented and supported API, simple scripts, or programming language.

Feature-rich integration

Flexible API customization to facilitate the use of all features supported by that security vendor's product— there are lots of functions (via API) that some security tools offer. Just because your tool is supported does not mean that all the functions are controllable via the security tool's APIs.

Additionally, if security tools have a lot of functions presented via API, it doesn't mean the SOAR tool can handle them all. For example, the firewall might only support adding an Internet Protocol (IP) address for blocking, and not a URL. A SOAR tool might not support requesting that a firewall return a response if it has seen a particular IP/URL/file hash.

Abstraction layer

Key to the value of SOAR tools is the availability of an abstraction layer so the analyst does not need to be an expert in specific APIs, scripts or programming language for specific tools. Rather, they can use logic and abstraction while the SOAR tool translates that into machine-specific API calls.

Source: Gartner (November 2017)

Automation

Some vendors use the terms "automation" and "orchestration" interchangeably as synonyms, although they are not the same concept.

Automation is a subset of orchestration. It allows multiple tasks (commonly called "playbooks") to execute numerous tasks on either partial or full elements of a security process. The security operations teams can build out relatively sophisticated processes with automation to improve accuracy and time to action. For example, a SIEM could check if an IP addresses has been seen, or block an IP address on a firewall or intrusion detection and prevention system (IDPS), or a URL on a secure web gateway. It can then create a ticket in your ticketing system or connect to Windows Active Directory, and lock or reset the password for a user's account.

Table 2 outlines the main requirements for automation in SOAR tools.

Table 2.   Summary of Automation Capabilities

Capability

Minimum Requirements

Process guidance

The ability to guide through standardized steps, instructions and decision-making workflow.

Workflow with multilevel automation

Flexible workflow formalization along with a set of predefined actions, as well as enforcement, status tracking and auditing capabilities.

The ability to automate workflows, with flexibility to inject human responses into the workflow.

Playbooks

The ability to code some playbooks, either using some standard language like Python or using some UI that helps the definition of playbooks.

Source: Gartner (November 2017)

Incident Management and Collaboration

Another function of the SOC that the SOAR tools make more efficient is the management of the incidents and the improved collaboration between team members working together on incidents.

This major function is complex. It deals with the life cycle of the incident from the moment an alert is generated, to the initial triage, to the validation of true/false positive, to the hunting and finally the remediation. To carry on this life cycle, the SOC team needs to collaborate and use an efficient collaboration framework, while threat intelligence becomes an integral part of the data points for this process.

Incident management and collaboration comprises several activities, described in the following sections.

Alert Processing and Triage

Two key metrics for information security are the mean time to detect (MTTD) and mean time to respond (MTTR). To accomplish an efficient incident response, SOC analysts need a better way to gather supporting information from a wide range of sources to assess and determine which alerts are real incidents. SOAR technologies gather and analyze various security data. The data is then made available and consumable by different stakeholders and for use cases beyond the original purpose. Triage will ensure that incidents based on information collected from other sources will be prioritized based on criticality and level of impact.

Event collection is commonly achieved through integration with a SIEM platform. Some solutions can automatically generate incidents for investigation. This removes the need to have a human first notice an incident and then invoke a manual step to create the instance of that incident. A key advantage of deploying SOAR technology is the first pass on alerts to reduce the noise or reduce the subsequent workload of analysts.

Journaling and Evidentiary Support

Some SOAR solutions can record information about actions taken, including details of the action itself, the person taking the action and when it occurred. Such journaling can be extremely useful in complex incidents where the following characteristics may apply:

  • There are questions as to whether apparently separate activity may or may not be linked to a broader operation by the adversary.

  • The incident takes place over an extended period, and so records of activity become a reliable corporate memory.

  • There are multiple people working on an incident or action

  • Regulations and other mandates require reports to be produced

Table 3 outlines the main requirements for journaling and evidentiary support in SOAR tools.

Table 3.   Journaling and Evidentiary Support

Capability

Minimum Requirements

User interface for investigation

Provide investigation timeline/screen to collect and store artifacts of the investigation for current and future analysis.

Help SOC analysts to continue the investigation/response during work shifts among analysts by keeping historical information of the incidents and notes.

Collaboration

Coordination of actions and decisions, particularly when easy communication is not possible (for example, due to time zone differences, work shifts or geographic dislocation).

Coordination of communication with other staff working in the same or related incidents for providing incident updates.

Source: Gartner (November 2017)

Case Management and Workflow

Two forms of security operations automation are often encountered: one focusing on automating the workflow and policy execution around security operations; the other automating the configuration of compensating controls and threat countermeasure implementation. To fully automate or semiautomate these tasks, solutions frequently provide libraries of common and best-practice playbooks, scripts and connectors covering remediation and response actions and processes. These should support the formalization, enforcement and gathering of key performance indicators of security policies. Custom workflow implementation must also be supported.

One of the biggest challenges in IT security operations capturing and retaining this "group knowledge" that exists within environments. Security operations staff often have an overabundance of notes, scripts and documents that describe in extreme detail how to perform a specific task. Additionally, these are often kept in an analyst's own head, and not fully documented. One of the hidden benefits of SOAR is the ability to codify tribal knowledge into tools, so it can be captured and used by many others. Gartner inquiries shows that workers tend to leave companies after about two to three years, on average. Turnover hurts security operations if key people leave and you no longer have access to institutional memory.

Table 4 outlines the main requirements for case management in SOAR tools.

Table 4.   Case Management

Capability

Minimum Requirements

Case management

Reconstructed timelines of actions taken and decisions made to provide up-to-date progress reports and to support post-incident reviews.

Collaboration and granular role-based access control and management

Exchange of information between teams, organization units and tiers.

Capturing knowledge base from security analysts

Build an internal knowledge base for incident resolution.

Leading products also provide a library of playbooks and processes for popular use cases, as well as access to a community of contributors.

Source: Gartner (November 2017)

Analytics and Incident Investigation Support

Proper investigation requires centralized tool that helps SOC analysts to quickly identify threats or incidents. During the process of investigation an ability to store artifacts will help through the identification and classification of threats. Those artifacts can also be used later to support further auditing demonstrating chronologically actions and data collected that resulted in a final response. The use of analytics will improve the reduction of false positive based on historical data and determination of level of risk assigned to incidents that will conduct the prioritization among many incidents.

Table 5 outlines the main requirements for analytics support in SOAR tools.

Table 5.   Analytics Support

Capability

Minimum Requirements

Incident investigation

Correlate incidents, including artifacts, to cross-match activity, and either view or link related incidents. The information should then be surfaced proactively to analysts.

Use forensics to perform a detailed analysis of activity that occurred before and after a security breach.

Source: Gartner (November 2017)

Management of Threat Intelligence

Threat intelligence is becoming a significant resource for detecting, diagnosing and treating imminent or active threats (see "Market Guide for Security Threat Intelligence Products and Services" ). Most SOAR tools, like many others in the security market today, include various forms of threat intelligence integration for this purpose. Some are built in, and others are able to be augmented by tools like a TIP. SOAR tools, however, allow not just themselves, but other deployed technology, to make use of third-party sources of intelligence. This can come in various forms: open source; industry leaders; coordinated response organizations, such as Computer Emergency Response Teams (CERTs); and a large number of commercial threat intelligence providers.

TIPs specialize in enabling intelligence-led initiatives in a security program as their base feature set. Today, they offer a sophisticated method for collecting and aggregating threat intelligence for use in security operations. They also have connections to existing tools, such as SIEM, firewall, secure web gateway (SWG), IDPS and endpoint detection and response (EDR).

Dashboards and Reporting

SOAR tools are expected to generate reports and dashboards for at least three classes of persona: analyst, SOC director and chief information security officer (CISO).

Because SOAR tools orchestrate incident response, have bidirectional communication with many other tools in the organization, and empower analysts, they are generating and accessing a lot of very valuable metrics that can be used for several types of reporting.

Table 6 outlines the main requirements for dashboards and reporting in SOAR tools.

Table 6.   Dashboard and Reporting Capabilities

Capability

Minimum Requirements

Analyst-level reporting

Report on activity for each analyst on metrics such as:

  • Number and types of incidents touched, closed and open

  • Average and mean time for each of the phases of the incident response; for example, incident and triage.

SOC director-level reporting

Report on the efficiency and behavior of the SOC on metrics such as:

  • Number of analysts; number of incidents per analyst.

  • Average and mean time for each of the phases of the incident response; for example, incident and triage.

CISO-level reporting

Report on priorities determined by business context metrics, such as:

  • Risk management: Demonstrate alignment of risks and IT metrics that would have a logical impact on business performance due to lack of controls, impact of incidents and regulations.

  • Efficiency: Demonstrate some level of cost reduction by minimizing incident impact. Key metrics would be MTTD, MTTR and reduction of labor time through automation.

Source: Gartner (November 2017)

Benefits and Uses

SOAR supports multiple activities for security operations decision making, such as:

  • Prioritizing security operations activities: Prioritized and managed remediation based on business context is the main target of security operations.

  • Formalizing triage and incident response: Security operations teams must be consistent in their response to incident and threats. They must also follow best practices, provide an audit trail and be measurable against business objectives.

  • Automating containment workflows: This offers SOC teams the ability to automate most of the activities to isolate/contain security incidents to be conceived by the human decision for the final steps to finalize the incident response.

Adoption Rate

Gartner estimates that today less than 1% of large enterprises currently use SOAR technologies. Higher adoption will be driven by pressing staff shortages, a relentless threat landscape, increasing internal and externally mandated compliance rules (such as mandatory breach disclosure), and a steady growth of APIs in security products. Also, the potential market for SOAR today is large organizations, with managed security service providers (MSSPs) as the primary target. Over time, smaller teams facing the same security threat problems will also begin to adopt SOAR tools. The ongoing skills and expertise shortage and the increasing escalation in threat activity will hasten the move to orchestration and automation of SOC activities.

Risks

Key risks for implementing SOAR include:

  • Market direction: In the longer term, adjacent technologies that are much larger and also focus on security operations (such as SIEM or other threat-focused vendors/segments) are likely to add SOAR-like capabilities. This will be sped up by acquisitions of SOAR tool vendors (for example, IBM acquiring Resilient Systems; Microsoft acquiring Hexadite; FireEye acquiring Invotas; ServiceNOW acquiring BrightPoint Security).

  • Limited integration value: Clients will not be able to leverage a SOAR tool if they lack a minimum set of security solutions in place to provide enough information to make a decision nor automating security tasks. For example, SIEM is often a key piece of technology for the use of SOAR tools due to its complimentary nature. Today, SOAR is most viable for Type A and Type B organizations. 2

  • Budget: Clients that are budget-constrained need to juggle conflicting needs of stretched budgets for all of IT, let alone security. They will likely not be early consumers of these technologies and instead will look to invest in more foundational security measures.

Recommendations

IT security leaders should consider SOAR tools in their security operations to meet the following goals.

Improve Security Operations Efficiency and Efficacy

SOAR tools offer a way to move through a task, from steps A to Z. For example, if a process takes an hour or two to perform, having a way to reduce that to 15 minutes offers a significant improvement in productivity. This is beneficial because:

  • Performing the task faster equals better time to resolution. The longer an issue is left unaddressed, the worse it can become, leaving the organization in a potentially risky situation for longer periods of time. Ransomware, for example, is a threat that can get exponentially worse with time.

  • Staff shortages are a critical issue for many organizations. The ability to handle processes more efficiently means that security analysts can spend less time with each incident and will thus be able to handle more incidents, allowing response to more incidents despite fewer resources being available.

  • Automation and orchestration allow your tools to work together to solve issues, versus operating in isolation with no context, which requires a lot of manual work to perform required tasks.

Product Selection

Security and risk management leaders should favor SOAR solutions that:

  • Allow orchestration of a rich set of different security (and nonsecurity) technologies, with a focus on the specific solutions that are already deployed or about to be deployed in an organization.

  • Promote an easy integration of tools not included in the out-of-the-box integration list.

  • Offer the capability to easily code an organization's existing playbooks that the tool can then automate, either via an intuitive UI and/or via a simple script.

  • Optimize the collaboration of analysts in the SOC; for example, with a chat or IM framework that make analysts' communication more efficient, or with the ability to work together on complex cases.

  • Have a pricing cost that is aligned with the needs of the organization and that is predictable. Avoid pricing structures based on the volume of data managed by the tool, or based on the number of playbooks that are run per month, as these metrics carry an automatic penalty for more frequent use of the solution.

  • Offer flexibility in the deployment and hosting of the solution, either in the cloud, on-premises or a hybrid of these, to accommodate organizations' security policies and privacy considerations, or organizations' cloud-first initiatives.

Better Prioritize the Focus of Security Operations

Prioritization is perennially a key problem. Favor SOAR tools that can help select the top 10 things to be doing today if you have 100 you can potentially do. Efficiency will not fix poor prioritization. SOAR tools can help with this by using external context, like threat intelligence, to help drive processes that have more context so that better decisions can be made in security operations. The goal is working smarter, not harder.

Don't "Boil the Ocean" — Focus on Critical Security Processes and Use Tools Such as SOAR to Evolve From There

Security teams are regularly tasked with fixing all things, all the time, 24/7, everywhere — but with the same budget and staffing as last year. This is clearly untenable, yet is a persistent observation we have with security operations teams in client inquiries. For security operations, we recommend focusing on executing well on key incident response processes, such as malware outbreak, data exfiltration and phishing. Focus on processes to address these types of situations very well, and then use this well-executed base to expand into other areas.

Representative Vendors

Anomali

Ayehu

CyberSponse

Demisto

DFLabs

EclecticIQ

IBM (Resilient Systems)

Microsoft (Hexadite)

Phantom

Resolve Systems

ServiceNow Security Operations

Siemplify

Swimlane

Syncurity

ThreatConnect

ThreatQuotient

Evidence

1 M-Trends showed MTTD reduced from 146 to 99 days between 2016 and 2017. See FireEye, M-Trends Reports, "M-Trends 2017."

2 Enterprise Types A, B and C definitions