Research from Gartner

Ten Innovations That Will Transform Hyperconvergence

To unify the core, edge, cloud and IoT, future hyperconverged infrastructure must be built on collective intelligence. I&O leaders looking to optimize data center functions must architect and implement more autonomous systems that communicate together to achieve defined, goal-based outcomes.

Key Findings

• Due to the growth of cloud, edge and the Internet of Things (IoT), the number of intelligent devices will grow at a 32% compound annual growth rate from 2016 through 2021, reaching an installed base of 25.1 billion units.
• Intelligent devices and systems often exist in silos, without the intelligent communication between them to create a common goal and purpose.
• To date, hyperconverged systems have tended to be narrowly configured and scaled, restricting their deployment as data center mission-critical systems.

Recommendations

Infrastructure and operations leaders focused on modernization and agility should:

• Prioritize working with vendors that add intelligent layers to their hyperconverged software-defined infrastructure (SDI) solutions, and not just to storage or networking.
• Develop APIs and connectors that allow smart devices, IoT and applications to communicate with each other and achieve common, goal-oriented outcomes.
• Start with smaller, achievable initiatives by first identifying the most pressing pain points in your infrastructure before applying your experience to larger, more connected initiatives.

Strategic Planning Assumption

By 2020, 30% of IT organizations that fail to apply AI and ML will cease to be viable against cloud or startups.

Analysis

The future of hyperconvergence will be built on a bedrock of intelligence. However, despite being embedded as a transformative technology within data centers, to date, the impact of hyperconverged infrastructure (HCI) has been limited due to the siloed nature of its deployments. Meanwhile, advances in artificial intelligence (AI) and machine learning (ML) are significantly increasing the IQ of distributed and integrated systems, while shallow learning enhances infrastructure intelligence. Additionally, traditional data center operations are inexorably moving beyond the core to encompass the edge, cloud and IoT.

This confluence of factors is making it increasingly important for infrastructure and operations (I&O) leaders to optimize their infrastructure by leveraging and increasing the available intelligence. Driven by the demand for consumer information, billions of external points of intelligence are creating exponentially greater amounts of data. This data comes from retailers, cars and even clothes; but it must be selectively processed if organizations are to extract its embedded business value. Given their highly integrated nature, HCI and hyperconverged integrated system (HCIS) platforms could be well-suited to the task. However, to engender the levels of agility required to improve resilience, reduce noise, and harmonize the workflow in the data center, cloud, IoT and edge, HCI functionality will need to evolve beyond its current rigid, appliancelike architectures. These systems are, by their nature, static. Thus, instead of deploying individual hyperconverged applications on one integrated architecture, it will become necessary to aggregate a number of appliances into one "superconverged" solution established with self-organizing principles of adaptation and resilience to common goals.

Gartner believes that future HCI will trend in this direction, and we have identified 10 innovations that will transform HCI for broader IT roles and use cases in the next five years and beyond (see Figure 1). We also believe that future HCI will need to subsume these innovations if it is to help meet organizational requirements for digital transformation.

Figure 1. Ten Innovations That Will Transform HCI

Source: Gartner (January 2018)

Innovation No. 1: The Evolution Beyond AI Toward Collective Intelligence

Intelligence will be everywhere in the near future. However, AI does not exist as a single application. Rather, it comprises a suite of applications and code that optimize different application segments within a workflow framework. Unless machines can communicate intelligently, the transformative power of AI in the data center and beyond will remain largely untapped. While there are HCI systems that include some form of machine learning (such as workload optimization or scheduling), no system or service is yet aggregating all the potential of cumulative intelligence from AI, ML and deep learning (DL) to create an ultimately aggregated, collective intelligence among HCI.

Just as society often relies on collaborative teams to maximize intelligent outcomes, HCI architectures require more than individual systems operating independently to achieve the superintelligence required for autonomous computing. However, most data centers are currently built around independent systems operating more or less in a siloed manner. Every system may perform its job well, but the components may not talk to each other or understand suboptimal behaviors. Smart data centers operating intelligent HCI require multiple communicating systems of intelligences. This will require improved levels of connectivity, data monitoring and analysis, with intelligent insights on performance and behavior. Since the technology is still quite nascent, I&O leaders looking to leverage collective intelligence should not try to solve broad, infrastructurewide issues at once. Rather, they should evaluate vendors on the basis of how they identify and utilize intelligent HCI to solve the most pressing pain points. I&O leaders should then leverage the experience to apply the same practices and principles to a broader and more strategic infrastructural scope.

Innovation No. 2: Autonomic Goal-Based Outcomes

Using APIs to create an automated workflow is only one of the strands required to create an intelligent data center. Equally important is setting goals and purposes. This is the essence of goal-based optimization, where parts A, B, C and D are synchronized in performance to achieve set goals. All systems in aggregate should understand the goal-based requirements and be optimized to achieve them. Intelligent hyperconverged architecture must optimize systems A, B, C and D toward a common purpose under an AI substrate, rather than work at cross-purposes.

Consider a driverless car. Every system must cooperate to achieve a common goal of getting the passenger from point A to point B safely and expeditiously. Rather than a single point of intelligence, there are multiple points of intelligence in navigation, visual pattern recognition, sounds and touch. Every sensor and every system is operating harmoniously to deliver aggregated intelligence that is more than the sum of its parts.

Innovation No. 3: Self-Organizing Systems of Intelligence

The eventual goal of intelligence and HCI is to deliver self-organizing systems of intelligence (SOIs). SOIs will be defined by their ability to utilize ML algorithms that continually monitor, process and organize system behavior from pools of data. In doing so, they will be in a position to maximize outcomes designed around common goals.

More than acting as just a monitoring system, SOIs must be able to identify vulnerabilities (including part wear, security holes and capacity erosion) and strengths to proactively learn from past situations. They must also be able to synchronize actions to preset goals, and self-organize for optimum behavior among networked participants.

Though data center infrastructure will still need to evolve for HCI as SOI, data center optimization is currently trending in this direction, and the pace of progression is increasing (see Figure 2).

Figure 2. The Evolution of Systems of Intelligence

Source: Gartner (January 2018)

As the chart shows, from the year 2000 through today, there was only a gradual evolution in infrastructure optimization, from blade consolidation through converged infrastructure to the adoption of HCI as a hardware-based appliancelike system. However, there was a marked upward tilt with the arrival of SDI. This is expected to accelerate with the advent of disaggregated, composable, intelligent fabric-based systems. As HCI becomes a well-accepted and widely-adopted form of infrastructure, new paradigms such as HCI as a service (HCIaaS) will increasingly emerge. Crucial to this evolution will be the need to continue adding layers of intelligence throughout the system.

Innovation No. 4: Digital Disaggregation and Integration

Disaggregation has been key to many of the highest-profile technological advances in the consumer sphere. For example, the smartphone could not have come about without disaggregating the camera, data, storage and voice functions from monolithic architectures and then reintegrating the functions into a single device. The same process can apply to the next generation of intelligent HCI. Initial HCI systems were sold as appliances, with one specific purpose, such as for virtual desktop infrastructure (VDI). However, modern HCI is potentially an aggregator of resources built around a hyperconverged cluster of nodes. These can be assembled, aggregated and composed into an integrated operating model. So rather than operating a number of silos, the disaggregated pieces can be assembled together to create a harmonious, integrated solution.

This trend will represent a significant shift for HCI. Previously, converged infrastructures have tended to be designed from the top down. The solutions were simple and effective, but they only did one job. It was difficult, if not impossible, to link the various applications. However, the disaggregated and integrated model will allow enterprises to assemble better, more agile devices that integrate every part of the system.

Innovation No. 5: Diagonal Siloless Scalability

Due to the monolithic nature of most HCI deployments, the hyperconverged paradigm has been based mostly on horizontal, cluster-based scaling, where enterprises scale their systems by adding nodes in parallel. At the same time, vertical scaling is enabled by more network-agile n-tier architectures, which are more widely used for mission-critical database applications. Thus, many architectures are composed of blocks that scale vertically and HCI clusters that scale horizontally. These block and HCI systems rarely integrate holistically, and hence workloads tend to become siloed.

However, the benefits of disaggregation mean that it is possible to achieve both vertical and horizontal scaling with HCI at the storage, networking and compute subsystem layers — hence, diagonal scaling that crosses any potential silos. There is no artificial restriction on applications due to HCI's architectural limitations, nor are there any restrictions on how applications can function together. Such an architecture could facilitate mixed workloads much more easily than horizontal-only architectures. It could also open the system up to new applications with strong quality of service (QoS) performance that were previously considered inappropriate for HCI.

Innovation No. 6: Distributed IoT-Based HCI

While disaggregation can facilitate diagonal scaling, the proliferation of IoT devices will dramatically alter the horizontal scaling paradigm of HCI. This is due to the massive increase in the number of devices that will be attached to HCI systems at the edge. For example, mobile devices and smart devices will feed information either directly into the data center, or into a cloud-based hyperconverged infrastructure. Because of these, new HCI systems will need to be architected to connect with an exponentially growing number of IoT-based devices. However, they will also need to be able to provide the vertical scaling required to process such a dramatic increase in data.

Innovation No. 7: Hybrid Cloud Intelligence

Though cloud-based infrastructures provide enterprises with greater elasticity, not every application or workload can or will be moved to the cloud. However, even on-premises HCI systems can utilize a cloudlike model to take advantage of the cloud's greater elasticity, in addition to its management and resource-gathering capabilities. By implementing a hybrid cloud intelligence model, organizations can access more memory, storage or compute resources as needed. But they can also create an intelligent management plane that offers an overall logical view of the entire end-to-end system utilization, whether that is in the cloud, on-premises or distributed at the edge. Such a model also allows for greater control and visibility of issues such as security, policies and incident management. This will be a crucial factor going forward, as current AI may lack the level of intelligence required to address issues across the broader ecosystem and distributed functionality created by the edge and IoT. Intelligence that was once focused within the physical data center will have to expand outward in tandem with the system. Adopting a cloudlike model will allow intelligent HCI to reach further across the ecosystem, while providing resources on a "just-in-time" and "just-enough" basis as the applications require.

Innovation No. 8: Consumption-Based Usage Models

A consumption-based model is an operating expenditure (opex) pricing model for on-premises data center systems with bidirectional scaling, which is paid on a periodic basis based on measured resource usage. Though this model is by no means new, in the absence of common standards, each vendor may have a different definition of what "consumption-based" means. Thus, there is as yet no consensus regarding the optimal applications, workloads or deployment models for such a deployment. Nevertheless, the shift in HCI toward a consumption-based usage model will fundamentally alter how HCI systems are optimized, designed and deployed.

Key benefits of consumption-based models include their elasticity and scalability. These are important considerations as IT as a service (ITaaS) and DevOps need access to resources in a just-in-time manner at just-enough costs. By combining predictive capacity automation with usage-based consumption, the resource allocation and availability should operate with little to no hands-on administration. Additionally, the intelligence should use data analytics of the workload type to compose the resources required. As an analogy, as the human body has a parasympathetic nervous system that adjusts its functions depending on its exposure to environmental elements, the AI in modern HCIs can optimize resource management through a usage-based consumption model. The nervous system does not need the conscious mind to tell it when to breathe. Similarly, longer-term HCI developments, with the help of AI/ML and neural networks, can operate as in a manner similar to a breathing, organic system. If the systems can eventually get to such a state (perhaps within five to 10 years), there will be no need for human intervention in resource allocation and management. However, until then, HCI will still require humans to make decisions.

Innovation No. 9: Human-Machine Orchestration

AI can help to advise on resource consumption at the appliance level, whether on-premises, through the IoT or in the cloud. However, deciding on the most appropriate use of resources still requires some level of human intervention, such as setting goals and monitoring usage accordingly. I&O leaders need to ask questions such as where to run applications, what happens when the app is not being used and how to maintain fluctuating amplitude in a nonvolatile manner.

Despite the need for human intervention, it is important that new HCI deployments differentiate the roles of AI-as-machine-operator and the human operator. For example:

• Does the system have full autonomy to make important decisions?
• Are there AI or ML tools that are wholly capable of making these decisions?
• Can the end users trust the machines enough to hand over the decision-making process, albeit with an override function?

These questions require solutions that can fundamentally alter decisions around trust, safety and security. This is why it is likely that the new management paradigm will operate as a collaboration between humans and machines. Business objectives of HCI are continually optimized. Thus, both humans and machines need to form a collaborative interface that helps to achieve defined, preset and adjustable goals. Under this model, the machine will make its recommendations, but the human will need to evaluate the recommendation and make a decision. Unlike the parasympathetic nervous system analog, this model demands human intervention in collaboration with AI directives to make sure that systems remain continually stable and continue to meet their goal-optimized needs.

Innovation No. 10: Composable Resource Management

A number of factors are coalescing to create a composable paradigm for HCI. Assemblage, disaggregation, distributed IoT and diagonal silos are all factors that feed into the composable resource management model. This allows application developers to assemble just the resources they need in terms of storage, networking and compute. However, underpinning the entire paradigm must be a failure mode. This means that if the app does not work properly, or if it needs to be redefined, the system can provide more memory, less compute or better networking as required. "Composable" basically defines the ability to look at the whole pool of resources available to individual developers, exposing the entirety of the available resources to their needs. HCI is showing signs of moving in this direction, where developers can assemble and disassemble the resources as needed for just-in-time, just-enough purposes.

Conclusions

The purpose of this document has been to provide a context and vision by which I&O leaders, data center operators, administrators, designers and architects, in conjunction with the CIO, can begin to plan for digital transformation. It will also help them to manage the increased complexity that data centers will face over the next five years and beyond. We have stressed the need for vendors and IT organizations to understand the growing needs and potential disruptions facing data centers and their operators. Without increased levels of intelligence, the span of this transformation will stretch far beyond human capabilities to manage and control. Whereas some capabilities will be further out in time, others are already making their way into systems and data center operations.The enterprise must be vigilant and explore leading-edge technologies, and be prepared to address current and anticipated pain points in an incremental progression.

Laying back is not an option. The longer the procrastination, the more likely it is that aggressive organizations will leapfrog laggards. The very viability of the enterprise may depend on the effort to begin preparing for the digital transformation. We reiterate our strategic planning assumption: By 2020, 30% of IT organizations that fail to apply AI and ML will cease to be viable against cloud or startups.

Source: Gartner Research Note G00349551, George J. Weiss, 15 January 2018