Emerging storage hardware and software enable I&O leaders to lower acquisition costs per terabyte and improve manageability. In addition to focusing on agility, automation and cost reductions, IT leaders should address the cultural changes and skill set shortages caused by digital business projects.
I&O leaders responsible for infrastructure modernization and agility can benefit from the following recommendations:
By 2020, the price of solid-state arrays (SSAs) will decline nearly 60% in terms of dollars per raw TB from the beginning of 2017.
By 2021, more than 30% of the storage capacity installed in enterprise data centers will be deployed with software-defined storage(SDS) or hyperconverged integrated system (HCIS) architectures, based on x86 commodity hardware systems, which is an increase from 10% today.
By 2021, more than 80% of enterprise unstructured data will be stored in scale-out file system and object storage systems in enterprise and cloud data centers, which is an increase from 30% today.
By 2020, 30% of organizations will leverage backup for more than just operational recovery (e.g., disaster recovery, test/development and DevOps), which is an increase from less than 10% in 2016.
The following remain the defining forces that are reshaping the enterprise infrastructure and operations (I&O) landscape:
IT leaders have responded to these growing forces with initiatives that reflect the importance of these trends.
Unabated data growth continues to outstrip storage budgets and staff growth. However, the adoption of new technologies and consumption models is enabling organizations to become progressively more efficient every year. These technologies include SSAs, HCISs, converged data protection solutions, data-aware storage systems, distributed file and object systems, SDS, storage analytics tools, and cloud storage. For IT organizations to remain relevant as providers of services to business units, they must take full advantage of these new technologies, as well as embrace a service model, as opposed to a gate-keeper model, particularly for Mode-2 workloads that support business units.
Cost containment and improvements in staff productivity remain strategic objectives, because capacity demands always grow faster than storage budgets or staff resources, and digital businesses demand 24/7 availability, which can't be achieved without autonomic operation. Interest in SDS, simplified backup and the adoption of HCISs is driven mainly by expectations of cost savings and improvements in staff productivity.
Despite their higher acquisition price, SSAs are replacing midrange and high-end storage arrays, due to their superior performance and near-autonomic operations, which translate into improved staff productivity. HCIS systems generally lower ownership costs by changing vendor supply chains and productizing software change control. Despite these technology improvements, storage remains a constant cost challenge, because organizations have difficulty accurately predicting capacity growth and the total cost of storing, managing and protecting data for its owners and users.
Digital business is the creation of new business designs by leveraging new digital technologies to change the way to do business and make the business process more efficient. It could affect ways to engage and service customers, market and sell products and solutions, increase operational and supply chain efficiency, and improve employee productivity. The 2017 Gartner Global CIO Survey, with more than 2,200 responses, highlights the following trends related to digital business:
The implication of these two trends is that, more than ever, IT— including storage, backup and data management teams — needs to inform business units and end users about potential rewards and risks involved with infrastructure as a service (IaaS) and SaaS projects that are out of IT's budgetary control. Business units and C-level executives may not be aware of potential hidden risks, so it's imperative to engage C-level executives and promote collaborated decisions between business units and IT. Moreover, for on-premises solutions, it's equally important that storage, backup and data governance products support business and user self-services and, at the same time, be placed under overall secure IT management policies to avoid abuse.
Business intelligence (BI)/analytics has been consistently ranked as a top investment focus for digital business in past Gartner CIO surveys. BI/analytics applications may require different storage platforms, data sources, repositories and retention policies. Hence, storage and data management teams need a clear understanding of the infrastructure impact of their digital business and associated business requirements.
Figure 1 highlights key trends that will help storage managers understand and prepare for the upcoming changes — from technology, management and skill set perspectives. The roadmap covers the key gaps between the current and future state of storage technology and management, and it provides a migration plan for a successful journey to the future.
Source: Gartner (March 2017)
The following scenario describes an idealized future state in storage infrastructure evolution. Rather than waiting for a top-down grand vision to be realized, Gartner recommends opportunistically adopting new technologies that deliver improvements in performance, agility, ease of use and cost. This pragmatic approach provides users with real-world experience to influence any vision for future infrastructure upgrades. Storage technologies on this shortlist include solid-state, cloud, HCIS, software-defined and, potentially, open-source solutions.
The top-down vision of a single-vendor storage infrastructure, with its promise of simplified management, disaster recovery and lower total cost of ownership (TCO), is being replaced by the growing awareness that it is problematic and costly to achieve. Increasingly, users identify application segments in which vendor consolidation can occur, while embracing a federated and disaggregated approach between segments that deploys the optimum technology at the right time and cost. One technology that promises this change is SDS. This is obtainable with SDS point solutions for subsets of workloads, but not across the entire data center. In some offerings, this approach includes self-orchestration and transparent data mobility among dissimilar storage arrays, but also acknowledges several realities:
Differences in storage availability are gradually becoming less significant. Storage array purchase costs per TB have dropped significantly, while performance, reliability and ease of use have improved, pushing performance constraints away from the storage array and into the network. Storage has become more agile and easier to scale, with less need for low-level device administration. Although storage will persist in distributed locations, management and monitoring will increasingly move to the central data center, as organizations will neither desire nor have the ability to recruit technical staff to provide on-premises support of storage in distributed locations.
In data center storage deployments, eliminating availability and scalability as primary evaluation criteria increases the importance of agility and extensibility when selecting the optimal storage solution. On-premises data center deployments will continue as organizations implement a service-oriented operating vision, leveraging technologies that deliver simplified storage management, more-autonomic operation and a self-service provisioning model. Cloud extensions will become standard requirements to support the reduction of on-premises storage and the ability to rapidly burst capacity, as well as avoid the overprovisioning associated with seasonal capacity fluctuations.
Extensibility is the ability to realize infrastructure visions by incorporating new technologies with minimal disruptions to policies, procedures and existing infrastructure. An agile, extensible and cost-effective infrastructure will have the following attributes:
Most users of public cloud storage services have been using and will continue to use cloud-native applications. Hybrid cloud solutions that bridge an enterprise data center with the public cloud have significant limitations in terms of performance at scale and cloud inefficiencies when retrofitting traditional storage and data management solutions designed for on-premises deployments. The cost and complexities of extracting data from the public cloud have also been inhibitors of cloud storage deployment. This is evidenced by the low adoption of cloud storage gateways and the slow adoption of cloud storage targets through enterprise backup and recovery software. However, hybrid cloud adoption could increase as network throughput increases at affordable prices and workloads become capable of seamlessly moving between on-premises and the public cloud with forthcoming solutions such as that from Amazon Web Services (AWS), Microsoft Azure and VMware.
With the availability of SSD-based storage options in the cloud and advances in encryption key management, enterprises are increasingly deploying mission-critical applications to the public cloud. As enterprise adoption of public cloud matures, so will the offerings from vendors to match performance and price expectations:
Digital businesses will develop new, bimodal, IT infrastructure operating procedures to facilitate data sharing between Mode-1 and Mode-2 organizations. This will improve agility, empower sales and marketing, and enhance business competitiveness. IT leaders are driving this evolution with investments in private and public cloud infrastructures and the creation of tiger teams that support Mode-2 applications. A private cloud infrastructure that mimics a public cloud infrastructure, such as the upcoming Microsoft Azure Stack, will drastically reduce the learning curve of the public cloud and accelerate private and hybrid cloud adoption for digital business initiatives.
Tiger teams should include individuals with diverse skill sets (including storage and data management), who can cooperate to implement a business vision. These teams need project managers who understand application needs and public cloud pricing models, as well as storage architects, who understand private and public cloud technologies, software-defined and HCIS solutions, open-source software (OSS), and open-source hardware.
Data and storage management are different competencies that are often conflated. Data management centers on retention policies, data deletion and archiving, backup, file analysis, converged data management, policy enforcement, and enterprise information management. Storage management centers on capacity planning, reporting, performance monitoring and troubleshooting performance problems, and negotiating RPOs and RTOs. Most storage array administration tools are proprietary, and have unique graphical user interfaces (GUIs) and reporting capabilities.
Storage management tools that provide asset management and forecasting assisted by analytics at a strategic level are severely lacking. Many infrastructure SDS solutions are merely software-based instantiations of their array counterparts. Data gravity restricts much of the appeal of management SDS to self-provisioning, because data migration can be error-prone. Open-source management frameworks (e.g., OpenStack) rarely eliminate the need for vendor-supplied array administration tools. As a result, it's difficult to compare storage arrays, storage networks and storage administration products.
Common characteristics of today's storage management and data management include:
Pricing for public cloud storage services have become increasingly complex, as cloud providers expand the differentiated services they offer and tweak pricing models for competitive advantage and additional revenue. The leaders in the IaaS market have set this new course to complexity, and other vendors have followed suit. Customers need to manage multiple variable costs to adequately determine pricing. Many of these variable costs — such as those associated with data egress, outbound bandwidth consumption, content delivery network usage, government cloud for additional isolation and security — are not typically considerations for on-premises storage acquisitions. Although cloud service providers are developing the means for automated selection of storage classes of object, file system and block storage, sizing a workload accounting for many variable cost factors is complicated.
The 2017 Gartner CIO Survey shows that 43% of organizations have adopted bimodal IT, which is an increase from the 38% in the prior year's survey. Although most bimodal organizations have realized clear benefits (such as "closer engagement between business and IT," "more innovation," "improved business perception of IT," and "faster time to market"), they also acknowledge the cost of bimodal IT (e.g., "technical debt," "additional head count to run Mode 2 program," and a "culture of 'haves' and 'have nots'").
Although the survey results show an encouraging trend, many traditional I&O organizations slow the development of Mode 2 business projects by forcing them to compete for scarce resources against the maintenance of existing projects and applications that consume the bulk of IT budget and staff resources. As a result, some business units, after procuring budgets for their projects, are starting to make their own IT infrastructure decisions, such as adopting cloud and SaaS offerings to replace the need for on-premises storage and backup, or even deploying their own appliances on their sites with little consideration of corporate policies and governance.
Because business units often overlook key aspects in which I&O leaders are well-versed (e.g., data security, governance and service management), Gartner strongly encourages a closer relationship between business units and IT regarding storage and other infrastructure solutions.
On-premises and public cloud storage infrastructures are, with few exceptions, managed via vendor-provided proprietary management tools and frameworks. The exceptions use open-source management tools, such as OpenStack. Whether proprietary or open-source, SDS frameworks generally rely on proprietary vendor tools to configure the storage arrays and troubleshoot performance problems. The benefits of these management SDS frameworks include preserving self-provisioning portals, orchestration and chargeback mechanisms, as well as minimizing the retraining and revisions of policies, procedures, and scripts that would otherwise accompany a change in the physical storage infrastructure.
Although dependence on proprietary configuration tools adds complexity to storage management, the plethora of new arrays with cache-centric data flows and improved GUIs that simplify storage provisioning is improving storage administrator productivity. However, the reduction in the cost and complexity of deploying islands of storage are likely to lead to more islands of storage, inefficient enterprise-level storage utilization, and complex workflow and management processes.
Storage vendors — such as Dell EMC, Hitachi Data Systems, Hewlett Packard Enterprise (HPE), IBM and NetApp, along with a number of emerging vendors and VMware — have created infrastructure SDS solutions running on commodity X86 hardware. However, the management of these software instantiations of proprietary storage arrays remains essentially unchanged. Hence, Gartner expects HCISs, self-protecting systems and easy-to-scale file systems to be at the forefront of the storage simplification movement.
The macro effect of these technology trends is that the storage administrator's job is rapidly changing. It is now more important for storage administrators and specialists to become enterprise architects and to verify that all processes, such as backups, disaster recovery, file analysis and storage resiliency, have been successfully implemented and are operating successfully on a daily basis. Better tools are needed to manage metadata, which can describe content, reveal information about the business value of data, and lead to data rationalization and management. Storage managers should identify and leverage tools that work across multiple application types, support various content and storage environments (including the public cloud), and emphasize data reuse and productivity.
IaaS and SaaS providers are not usually transparent with their infrastructure designs, arguing that customers should focus on their applications, rather than the underlying cloud infrastructure. More importantly, the providers are not financially responsible if data is lost, so many provide only limited credit for the downtime. Users are responsible for recovering their own data, because many public cloud providers do not offer granular restore mechanisms with isolated, immutable recovery points.
The most significant gap that prevents effective enterprise use of public cloud storage services is related to the skills of IT staff. Many enterprises try to use IaaS offerings to become more agile with application development and delivery. However, enterprises often lack the skills required to employ the automation capabilities inherent in API-driven public cloud storage services. As a result, enterprises use public cloud IaaS as rented virtualization, which often has limited benefits.
According to the 2017 Gartner CIO Survey, organizations with different profiles have drastically different adoption rates of digital business and bimodal IT. Top performers tend to have much faster adoption than typical performers and trailing performers, especially in the following areas:
Gartner recommends an iterative migration plan, which will help I&O leaders establish a more connected storage environment, simplify storage management and establish greater overall alignment with the business. This plan is broken into short-term (higher-priority), midterm (medium-priority) and long-term (lower-priority) projects (see Figure 2). Priority corresponds to when a project will kick off, with the high-priority projects getting underway in 2017.
Source: Gartner (March 2017)
SSAs, hybrid storage arrays and integrated systems have shortened the ROIs associated with deploying these technologies in the data center. Although initial deployments may be tactical, when coupled with the availability of simplified data migration/conversion tools, these systems' scalability will often make them strategic investments. Users should follow these best practices:
Many enterprises adopt public cloud storage as part of a cloud-first application deployment initiative intended to bring agility to the delivery of IT services. However, they should be aware that cloud storage has a range of services with different pricing and performance levels, lacks meaningful availability guarantees, may require changes in security policies and procedures, and still requires storage management.
Factors to consider before moving to public cloud storage include:
IT departments need to fund the bimodal training of a small number of exceptional concierge staff to gain broad infrastructure knowledge, including public cloud and SaaS. This will help business units deliver on a project quickly and confidently. To create a nimble and capable tiger team in anticipation of new digital business initiatives, Gartner recommends that IT departments do the following:
First, acknowledge the differences in attributes, skills and attitudes needed to support Mode-1 and Mode-2 operations (see Table 1). Then, make the organizational changes required to create an innovative culture that rewards risk taking.
|Attributes||Mode 1||Mode 2|
|Architectural approach||Scale up||Scale out|
|Deployments||Packaged appliance-based; high software and hardware costs||Software-centric; industry-standard hardware; cloud-based|
|Storage attributes most valued||Reliability, availability and serviceability (RAS) and performance||Scalability and adaptability|
|Sourcing||Incumbent enterprise vendors; long-term asset management||Emerging vendors; short-term ROI|
|Organizational structure||Dedicated, vendor-certified storage professionals||DevOps-oriented; open-source/multiskilled professionals|
Source: Gartner (March 2017)
Before moving application data to public cloud storage, compare the advantages of adding or refreshing on-premises storage systems relative to public cloud storage solutions. Cloud storage migrations are not always successful, nor are they always less expensive than on-premises solutions. The staff resources needed to manage cloud provider relationships grow in proportion to the number of services being offered, the number and types of applications being moved, and the amount of data being migrated.
New technology storage systems, HCIS, software-defined infrastructures (SDI), and new delivery models (capacity on demand, hosting and colocation providers, and cloud-like utility models) are lowering storage acquisition and ownership costs, while improving agility and disaster recovery usability. Add the cost and risk avoidance associated with moving applications and/or their data to the cloud, and the decision to stay on-premises or go to the cloud becomes a legitimate question, based on your organization's priorities. Foremost among the expenses are application selection, achieving buy-in from all stakeholders and security-related changes. The risks include errors in forecasting bandwidth charges and storing data on a shared public infrastructure versus complete control of the entire hardware/software stack, more configuration flexibilities, better cost projection and, possibly, lower cost, relative to the public cloud storage.
Identify and train the skills needed to support Mode-1 and Mode-2 operations (see Table 2).
|Storage architects and engineering||Infrastructure/cloud architects|
|Storage management||Application architects
Site reliability engineers
|Backup and disaster recovery||Chief data officer|
|Asset management||Data scientists|
|Information life cycle management||Vendor/business liaison|
Source: Gartner (March 2017)
By supporting and improving the success rate of Mode-2 initiatives, IT can gain an opportunity to deliver value to the business. By deploying Mode-2 applications in the cloud, IT can also gain hands-on expertise with the public cloud and a better understanding of its strengths and weaknesses. These experiences will enable IT to better align its I&O vision with the needs of the business.
Turn this integration into infrastructure-level, optimized services. Practical experience with the financial and operational benefits and the risks of new on-premises technologies versus cloud storage will enhance IT's ability to deliver agile, cost-effective services to business units. It will also enable IT to debunk the myths surrounding the expanding use of the cloud IaaS, SaaS and HCIS solutions that are now gaining market share.
By embracing digital business initiatives, the IT organization can evolve from a cost center to a competitive advantage for the organization. IT can also influence the design of Mode-2 initiatives, rather than react to them, and gain the valuable new skills required for Mode 2.
Source: Gartner Research Note G00324339, Pushan Rinnen, John McArthur, Stanley Zaffos, Raj Bala, Dave Russell, Julia Palmer, Valdis Filks, Garth Landers, Santhosh Rao, Robert Rhame, Julian Tirsu, Shane Harris, 8 March 2017
Gartner annually conducts the industry's largest CIO survey. Approximately 2,600 CIOs in 93 countries — representing $292 billion in IT spending — were surveyed in October 2016. From this data, "infrastructure and the data center" has fallen behind "business intelligence and analytics" and "cloud services/solutions" from the prior year's No. 2 position ahead of cloud. However, these top three trends will continue to translate into greater demands from the storage infrastructure.
When it comes to infrastructure-specific challenges, the attendee polling results at the December 2016 Gartner Data Center, Infrastructure & Operations Management Conference showed many recurring themes — for the seventh straight year, "developing a private/public/hybrid cloud strategy" topped the attendees' greatest concern.
"Aligning I&O with the business" continued to be the second-most-important challenge, followed by "modernizing legacy applications," "finding/retaining IT talent" and "implementing bimodal IT." Among the top five challenges, only "modernizing legacy applications" is associated with Mode-1 activities, whereas the rest are related to Mode-2 activities — enabling business initiatives, including the digital business. Although some attendees of this conference were optimistic about their application portfolio being in Mode 2 by 2020 (42% of 83 attendees of a session believe half or more of their applications will be in Mode 2), many more (75% of 169 attendees of a session) told Gartner that "the culture change" will be the "hardest challenge to overcome moving toward 2020."