LICENSED FOR DISTRIBUTION

Integrating Security Into the DevSecOps Toolchain

Published: 16 November 2017 ID: G00334264

Analyst(s):

Summary

As DevOps gains popularity for rapid delivery and innovation of new IT-enabled capabilities, concerns about security and compliance increase. Security and risk management leaders must adapt security tools, processes and policies to the DevOps toolchain without slowing the development process.

Overview

Key Challenges

  • DevOps adoption is increasing as an alternative to traditional waterfall and agile development methodologies, but security and compliance are typically afterthoughts.

  • DevOps stresses automation to achieve scale, but security has traditionally been slow, process-heavy and gate-driven — the antithesis of automation, transparency and speed.

  • The majority of DevOps developers have no knowledge of security.

  • Traditional application security testing offerings weren't designed for speed and transparency.

  • For some applications in specific industries, new versions need to be government-recertified after every production update, making rapid change an issue.

Recommendations

Security and risk management leaders supporting improving application security DevOps teams should:

  • Work in consultation with project leads to decide upfront which projects are a good fit for DevSecOps, based on business risk and regulatory requirements.

  • Adapt existing secure development life cycle tools into the DevOps process by "shifting them left" into small actionable steps that developers can take quickly, and "shifting right" to automated tools and responses.

  • Require all application security testing vendors to fully API-enable their solutions for automation and, ideally, provide explicit support for integration with common continuous integration/continuous development (CI/CD) tools.

  • Improve and coordinate actionable feedback from the tools into the life cycle by targeting intelligence back into phases where it can be used and in the format the developer prefers.

Strategic Planning Assumption

By 2019, only 10% of DevOps initiatives will have achieved the level of security automation required to be considered fully DevSecOps, up from less than 5% in 2017.

Introduction

DevOps emphasizes the tearing down of walls between development and IT operations with a mutual goal of increasing the speed of delivery and continued innovation of IT-enabled capabilities. Gartner inquiries related to DevOps have increased steadily, from a few a month in 2015 to over 25 new calls per month in 2017. Both security and development leaders are working to adapt programs and processes to the methodology. While DevOps does promise a number of advantages to the business gained through rapid iteration and development, it also poses formidable challenges to security and risk management leaders charged with supporting secure development in those projects. To enable DevSecOps, security teams will need to adapt existing application security programs to match the speed of DevOps required while not unnecessarily sacrificing security or compliance.

Traditional application security testing relies on heavyweight, one-time gating inspections, typically performed during testing, taking days if not weeks and requiring security professionals to perform them. This won't work and it won't scale for DevSecOps. DevOps emphasizes continuous feedback and improved automation. Security needs to adopt and support a mindset where security starts at the very beginning of the service creation, throughout the DevOps processes, and is continuous, automated and improves with each subsequent iteration. Figure 1 shows Gartner's view of DevSecOps as a continuous cycle, with each phase flowing smoothly into the next (thus, the infinity loop metaphor). The emphasis is on a continuous delivery cycle with automated processes informing incremental improvements, security and monitoring.

Figure 1. DevSecOps: Integrating Security in DevOps
Research image courtesy of Gartner, Inc.

Source: Gartner (November 2017)

Perfect and invulnerable applications are not possible; thus, security and risk management leaders must balance the need for security with development's need for speed. Not every project should be converted to DevOps-style development. Tackling these issues is the critical dilemma with building out DevSecOps programs. Here, we provide best practices for building out a DevSecOps capability.

Analysis

Decide Early on If DevSecOps Is a Good Fit for a Particular Project

DevOps as a methodology places its emphasis on collaboration, speed, agility and iteration. However, some applications are bound within constraints around privacy and access, federal regulations, and mandated oversight. This makes changes to production much more cumbersome and binds development with process and compliance issues. Financial services, healthcare and manufacturing are some vertical industries where government regulation restricts what can be done in some parts of the production environment. For example: Pharmaceutical manufacturers need to have the FDA recertify their production environment after certain kinds of software updates. Since this is an expensive, time-consuming process that puts the production system offline for a period of time, these outages tend to be scheduled in advance during other downtime windows.

However, even in a highly regulated industry, not every project is subject to these kinds of restrictions. In fact, generally the number of restricted or regulated systems is a small percentage of the overall set of systems. DevSecOps is often quite suitable for rapid innovation and iteration at, for example, the customer-facing edge.

While DevOps can be particularly challenging for security and risk management leaders to support, every effort should be made to do so when the risk to the business is acceptable. It is incumbent upon security and risk management leaders to identify those systems upfront, then work with development to define parameters that can guide project leads when a DevOps approach may not be a good fit based on regulatory or security requirements. Acknowledge that there are good reasons for the business to employ DevOps, including faster speed to market and improved quality releases, and work to support those as much as possible. In the "State of DevOps Report," a survey by Gartner's CEB Research team reported that 39% of IT leaders found that DevOps helped meet business demands by a moderate amount, and 61% said it helped extensively. Emphasis for security and risk management leaders should be on how to make it work when possible. Find ways to facilitate secure development for those projects where it does not introduce unacceptable risk. No-go decisions should be aligned with the risk posed to the business and similarly communicated to stakeholders.

Recommendations:

  • Work in consultation with relevant stakeholder to define guidance on when DevSecOps may not be appropriate.

  • Match the development methodology to the project's constraints before assigning a team.

  • Highly regulated or legally mandated gates are poor matches for DevSecOps, but customer- and partner-facing interfaces are often good candidates.

Adapt Existing Secure Development Life Cycle Tools Into the DevOps Process by Spreading Them Throughout the Life Cycle

The phases of the DevSecOps process are outlined in "DevSecOps: How to Seamlessly Integrate Security Into DevOps," and, at a high level, can be broken down into 10 familiar categories (see Figure 2).

Figure 2. 10 Categories of DevSecOps Tooling
Research image courtesy of Gartner, Inc.

Source: Gartner (November 2017)

Many of the tools that have traditionally supported the secure software development life cycle (secure SDLC) will be applicable in DevSecOps, although the speed and frequency with which they run may be very different. In some cases, new tools will need to be purchased to address gaps or when the vendor doesn't modify its capabilities to be suitable for automation (for example, API enablement).

  • Shift Left: A common phrase used in the context of DevSecOps, "shift left" maps directly to a strategic approach to shift more testing to the left side of Figure 2, as early as possible in the product's life cycle. Security leaders should place the emphasis on technologies for early and automated detection, and into small actionable steps developers can take quickly, starting with planning.

  • 1-Plan: Planning is crucial, and it's important to address security and technical debt. Decide what security issues you want to address in the next few scrums and add them to the work list. Also examine what security requirements you will need to start addressing and if this is the time to start laying down the foundational code. Security training, threat modeling and tool training are also addressable during this phase (see "Formally Defining Security's Threat Management Program: Guidance for Managing Threats, Not Just Vulnerabilities" ). You may choose to stand down some percentage of your development staff for the next few scrums and have them run training while other parts of the team address other issues. Then, in the next planning cycle, change them out until all your developers are up to speed.

  • Immediate feedback is a key element of DevSecOps, and it's well established that the sooner a bug can be caught, the less expensive and easier it is to fix. In the next few stages of the life cycle, Gartner emphasizes a shift-left philosophy — that is, favoring tools that developers can use to get clear, actionable information and receive feedback early (i.e., further left) in the process.

  • We also recommend developing security champions, developers who have had some minimum security training and act as subject matter experts for answering basic security questions, spotting problems and acting as an interface with IT security (see "DevOps Security Champions Help Organizations Gain Leverage Without Training Everyone" ).

  • 2-Create: Emphasize a shift-left philosophy, and favor security tools and solutions that integrate directly within the developer's integrated development environment (IDE). Many security tool vendors provide a "security spell checker" that catches common security mistakes and offers pointers to instructions on the defect and the solution. See "Magic Quadrant for Application Security Testing" for an overview. However, be careful to avoid useless warnings or unhelpful advice that inadvertently trains developers to ignore the tool. In an effort to avoid these, IDE-based security testing tools will often need to sacrifice breadth to reduce false positives, and will not identify as many issues as a dedicated static analysis test of the whole system. IDE plug-ins are not a substitute for comprehensive static analysis testing, but, in the spirit of DevSecOps, they catch simple errors earlier on.

  • Other critical activities in the create phase include threat modeling, security architecture review (for new components or larger projects that span multiple iterations) and open-source review. All of these activities can be greatly facilitated by security champions.

  • 3-Verify: The DevSecOps scanning here can be broken into two major categories (1) scanning for known vulnerabilities and misconfigurations in known/reused code and (2) scanning for unknown vulnerabilities in custom code. In modern DevOps development, the majority of code in applications is "assembled" not developed, and the majority of the code is reused from open-source and third-party libraries, frameworks and components. Scanning these first for known vulnerabilities and known misconfigurations will address the majority of risk.

  • This is where software composition analysis (SCA) solutions with governance capabilities can be used to enforce the organization's open-source software (OSS) policy when components are downloaded, allowing verification to take place proactively and essentially creating a repository of "known good" components that developers can leverage at the onset of a project. This can then be automated for other projects.

  • For scanning of custom code, traditional static and dynamic application security tools (SAST and DAST) can be applied along with SCA to help understand your project's dependence on underlying open-source code (e.g., Struts). As DevOps becomes more widely adopted, SAST vendors are beginning to offer "fast scans," static analysis scans that only look for the most common mistakes, but complete in minutes instead of hours. WhiteHat Scout is a typical example. This doesn't eliminate the need for full scans, which should still be run periodically, but lighter, faster scans can be incorporated as part of a DevOps organization without the time impact of heavier products.

  • Interactive application security testing (IAST) is a key technology in the verify phase. IAST is an improved form of DAST where the application is instrumented from the inside while scanning is performed from the outside. IAST is performed on running code and can be integrated into the overall DevOps process. In IAST, code is instrumented so that when it is tested or attacked, it generates data that can be used to improve security or respond to the attack. Runtime application self-protection (RASP) is a shift-right technology resulting from IAST. While organizations have been reluctant to deploy RASP at runtime because of performance and compatibility concerns, these issues are less relevant during development (the left side of Figure 3).

Figure 3. IAST Is a Shift-Left Security Technology; RASP Is Shift Right
Research image courtesy of Gartner, Inc.

Source: Gartner (November 2017)

  • Note that most IAST implementations require instrumentation of the application runtime (Java, .NET, PHP, Ruby, Python, Node.js, etc.) and not all vendors support all platforms, so IAST may not be possible on all applications and more traditional DAST may need to be used (assuming it can be fully automated and performed transparently for developers).

  • The verification phase is also a good place to leverage security champions, developers who have had some security training. In your security champions' training, you can include some work in reading security testing tool results as part of their skill set. This can help to train your developers a little more, and the champ can usually distinguish between critical or high-severity issues and lower-priority problems (or false positives). They can also get a better feel for the overall security posture of the release, and raise any concerns or emphasize issues to be addressed in later scrums.

  • Other activities in the verify phase include protocol and input fuzzing, application vulnerability correlation (AVC), and mobile application security testing (MAST). "Hype Cycle for Application Security, 2017" has details on these and other application security technologies.

  • 4-Preproduction: Favor solutions like IAST that allow you to instrument code and see how it reacts to both known attacks and to fuzzing and Chaos-Monkey-style testing. The attacks are nondeterministic, so how much time you invest is up to you, but this is exactly the kind of testing to overcome testing bias and rigid thinking, and reflects the nature of actual zero-day events. When code has been deemed ready, it should be checked into the release branch and signed and time-stamped. This will help serve as an integrity check for release. Peach is a good example of an open-source nondeterministic testing tool.

  • Nondeterministic tests like input fuzzing represent a unique opportunity when applied to DevOps. Unlike conventional testing, there is no specific recommendation for how much of it to do because it's based on randomness. A nondeterministic test may run for weeks and never find a vulnerability, or it may hit one on its first try; there is no way of knowing in advance. However, because DevOps emphasizes iterative development, the code runs through this kind of testing far more often than code developed in a method like waterfall. Because random tests accumulate results iteratively, this effectively becomes much more testing over time.

  • 5-Release: Check the time stamp signature and deploy into CI/CD. We envision that multiple signatures will be used within DevSecOps environments, reflecting the successful completion of different types of testing — for example, a signature for SCA completion, another for IAST and another for complete vulnerability scanning. Code will not be allowed to be released into production without the confirmation that it has met the required security and compliance checks and policies.

  • Shift Right: Emphasize continuous monitoring and automated tools that require no human intervention. Just as in development, we emphasized a shift-left philosophy because errors were easier to correct sooner in the cycle. Now we move to a shift-right stance, shifting focus to the attack surface. We depend on technology to detect and protect applications from threats and for applications to "fail closed" protecting their data and where possible. We also want information about the existing network environment and want to feed information about attacks, attempted and successful, back into the process to help us inform the next cycle. Use this intelligence from applications about their environment to inform developers and turn it into actionable improvements in security, such as reprioritizing training, adding security requirements or augmenting the threat models based on this intelligence.

  • 6-Prevent: Focus on configuration assurance at instantiation — that the code is what we expect it to be and that it indeed meets all of the requirements to be released into production. A common security control will be application control/whitelisting, enforcing that the server workload is running what we expect it to be and is restricted to only running it, behaving in this way. This approach is foundational to workload security and should be considered the primary control for workload protection ( see "Market Guide for Cloud Workload Protection Platforms" ). Common defense in-depth (DND) measures (DDoS protection, etc.) also apply here. For network traffic, other technologies like identity detection and prevention systems (IDPSs), user and entity behavior analysis (UEBA) or increasingly next-generation firewalls (NGFWs) are able to observe the behavior of connections or users and determine intent and deploy active protections (see "Magic Quadrant for Enterprise Network Firewalls" ).

  • 7-Detect: Security leaders must acknowledge that not all vulnerabilities will be identified in development, and some will inevitably make it into production. This is true of any application development methodology — including DevOps — and, as such, irrespective of development approaches, systems need to be architected to rapidly detect issues in runtime. Here the applications depend on technologies like RASP, a broad range of technologies that allow the applications to respond when they are under attack, by either self-protecting or failing safe (see "Emerging Technology Analysis: Runtime Application Self-Protection" ).

  • For example, in addition to RASP:

    • Web application firewalls (WAF; see "Magic Quadrant for Web Application Firewalls" ), especially if the WAF is configured for protection based on explicit knowledge of known vulnerabilities gained through dynamic application security testing (see "Magic Quadrant for Application Security Testing" )

    • Application monitoring and protection (see "Application Performance Monitoring and Application Security Monitoring Are Converging" )

    • Botnet mitigation to protect against user interface abuse and information scraping

    • In-depth defense at the application layer, including load balancing, denial of service and DDoS protection.

  • See "Structuring Application Security Practices and Tools to Support DevOps and DevSecOps" for a guide to vendors and tools covering the IAST and RASP space.

  • 8-Respond: There are many existing application security response technologies, but they usually fall into three categories:

    1. Rely on existing defense in-depth to protect the applications. For example, many network hardware providers will offer DDoS protection as part of their router and switching hardware. This can be a fine way to respond to threats providing you've done the basic threat modeling to match the existing defenses to your application.

    2. Rely on more active technologies like RASP or WAF, which can detect the type of attack the application is experiencing and muster an appropriate defense and alert or record the attack. RASP is closely associated with IAST and your IAST testing should inform the kinds of responses that the application uses to defend itself successfully.

    3. Rely on more passive responses like application shielding, code obfuscation and API gateways.

  • In all cases, intelligence from the attack surface is critical to both mounting the right kind of defense and feeding information back to the development teams about how the application responds in the real world. It's essential to make sure there is a link, through security incident and event monitoring (SIEM), security orchestration or monitoring, to get information back to the plan phase and allow the developers to adjust the system. The response to that information may be in the form of incident response, operational adjustments or simple security technical debt. In any case, it needs to be incorporated back into the overall development flow.

  • 9-Predict: Using the visibility and telemetry received from the detect and respond phases, we want to be able to predict and anticipate what new countermeasures the system will need and build them. For example, unusual web path traversal patterns can indicate an underlying vulnerability that is being actively exploited by attackers and that needs to be addressed.

  • This requires the data to be collected and analyzed from the attack surface and fed back into the product planning cycle. This visibility and usage data can be leveraged to support threat modeling and security requirements gathering activities for similar subsequent projects as well. In addition, standards like STIX and TAXII are becoming increasingly recognized as a useful way for organizations to share intelligence among themselves and with law enforcement and government agencies.

  • Third-party threat feeds and intelligence also become a critical part of this phase — for example, when a new vulnerability is discovered in a widely used framework such as Struts. This information can be used to prioritize a workflow element into development to fix the vulnerability. If the vulnerability is critical and the application/service affected is critical, then development can prioritize a new automated release using the updated library or framework that resolves the issue.

  • 10-Adapt: Adapting security practices to DevOps will often start in stages, and improvements or gaps will inevitably be identified. The goal is to take all the information the system and its instrumentation have received and use it to inform the next round of requirements, both at the scrum level for critical-level events and at the planning level for less immediate issues. This may also involve some level of analytics or data reduction techniques to make the information into actionable activities. Application security testing solutions will need to be reconfigured and refined over time to better support programs. The sheer number of findings these solutions can produce will often overwhelm security and development teams, and solutions will need to be adapted. For example, SAST will need to be tuned to focus on higher-value findings and to suppress false positives. Continuous feedback and development should become a mantra for the security team as well, which will use the collected alerts and statistics, as well as feedback from stakeholders, to improve processes, tools and security practices to better support future projects.

Recommendations:

  • Plan: Threat modeling, security requirements gathering and security training

  • Create: IDE security plug-ins

  • Verify: SAST/DAST/SCA/IAST

  • Preproduction: Chaos Monkey, fuzz testing, code signing, IAST

  • Release: Signature verification, integrity checks

  • Prevent: Defense in-depth measures

  • Detect: RASP, penetration test, UEBA

  • Respond: RASP/WAF application shielding and obfuscation

  • Predict: CVA, IoC/TI STIX TAXII

  • Adapt: Technical debt/incident response (IR)

Evidence

Gartner's number of inquiries has grown steadily, month over month, since the introduction of the term "DevSecOps" in 2015. By the end of 2017, we are expecting to see over 25 new inquiries per month (see Figure 4).

Figure 4. Monthly Searches for the Term "DevSecOps" on g.com
Research image courtesy of Gartner, Inc.

Source: Gartner (November 2017)

Note 1
A Glossary of DevSecOps Terms and Acronyms

Table 1.   DevSecOps Terms and Acronyms

Abbreviation

Practice or Technology

Additional Notes

ADLM

application development life cycle management

Application Shielding

a set of technologies that is employed to add security functionality directly to applications for the active prevention and detection of application-level intrusions

It encompasses anti-tampering and obfuscation capabilities and intervenes in all three stages of prevention, detection and remediation.

AST

application security testing

  • Static — SAST

  • Dynamic — DAST

  • Interactive — IAST

  • Mobile — MAST

ASTO

application security testing orchestration

AVC

application vulnerability correlation

CI

continuous integration

CD

continuous delivery

Chaos Monkey

a nondeterministic testing methodology, similar to fuzzing, applied to applications

https://netflix.github.io/chaosmonkey/

CWPP

cloud workload protection platforms

CVA

correlated vulnerability analysis

A method for looking at patterns of vulnerabilities to find root causes

Fuzz testing

using variations of expected or random inputs to an application to test its resiliency

Tends to be nondeterministic, but similar to what actual hackers do.

IaC

infrastructure as code

IDE

integrated development environment

IDPS

identity detection and prevention system

Useful component of NGFWs for identifying and blocking malicious traffic

IoC

indicators of compromise

OSS

open-source software

RASP

runtime application self-protection

SCA

software composition analysis

SIEM

security incident and event monitoring

UEBA

user and entity behavior analysis

Useful for detecting anomalous behavior in applications or users.

WAF

web application firewall

A firewall that goes between the application and the rest of the network that can detect and respond to threats

Source: Gartner (November 2017)