The size, complexity, the distributed nature of data, the speed of action and the continuous intelligence required by digital business means that rigid and centralized architectures and tools break down. It is pushing the limits of manual approaches to data management, analytics, data science and machine learning and AI.
Virtually every aspect of data management and analytics content and application development and sharing of insights is incorporating ML and AI techniques to automate or augment manual tasks, analytic processes and human insight to action.
The imperative of building an agile, data-centric architecture that augments every aspect of the business, and responds to constant change, has never been more critical to business survival.
Intelligent data and analytics capabilities — enabled by more aggressive transition to the cloud — make emergent and agile data fabrics and explainable, transparent insights possible at scale. This is necessary to meet the new demands and expand adoption.
The path to success requires making the right choices in the face of unprecedented business demands and rapidly changing and complex technology options. We know that a lot of organizations working on data and analytics struggle with balancing between investments that drive innovation and renovating their technology core.
The very challenges created by digital disruption — too much data — have also created an unprecedented opportunity. Leveraging these new vast amounts of data, when coupled with increasingly powerful processing capabilities enabled by the cloud (for both data management and data science), makes it now possible to train and execute the algorithms at the very large scale necessary to realize the full potential of AI.
We are seeing a growth in the use and application of data science, machine learning and AI across all industries, particularly as AI matures and the democratization of AI accelerates. AI will no longer be for the privileged few. AI-enabled analytics and data management tools (what Gartner calls Augmented Analytics and Augmented Data Management) will empower the many to process the vast amounts of data needed for advanced analytics at scale. This will make it possible for a broader range of skills, such as those of the citizen data scientist and developer data scientists, to create AI-driven insights and embed them in applications used by everyone across the organization.
AI as a driver of how employees, customers and partners interact with all business applications will feature prominently in this transformation where natural language and conversational interfaces will also open up insights to more people in and beyond the organization.
Augmented and mixed reality, still in its infancy in the enterprise, will also start to play a role in the enterprise, beyond gaming and entertainment, including in data and analytics.
With so much data and insights from diverse sources, data storytelling — part art, part technology-enabled — will be a critical personal and organizational competency necessary to drive optimized actions across the enterprise.
Moreover, the broad distribution of analytics capabilities to everyone in the enterprise is giving less-technical individuals across the business the ability to generate value from data assets and analytics. To enable this growth, we see considerable adoption of capabilities such as agile data preparation, data cataloguing and metadata management technologies, and new adaptive data-governance-related processes. These are enabling content authors using any tool anywhere in the business to do more in creating and using trusted, high-value data. As more and more types of people have access to analytics, a strong data foundation is imperative now more than ever.
Organizations must consider how graph technologies enhance the accuracy of data science and machine learning and AI, and form the foundation of capabilities such as natural language processing as well as complex data modelling enabled by knowledge graphs. This will enable the organization to capture all of the knowledge about what data you have, where it resides, how it’s all related, who uses it, why, when and how — and then using that insight to provide more personalized, automated and properly governed solutions to the business.
New roles such as the data engineer and processes supporting DataOps and user communities are key enablers of this more distributed yet governed approach.
As machine learning and AI move beyond prototyping and early departmental deployments, success will be measured by the impact of models actually deployed in production — not just by the number of models created. It is the models that are operationalized that are driving measurable business impact.
MLOps and new AI governance processes, which include model and AI operationalization, management and explanability will be critical capabilities to understand and scale the value of these investments and their contribution to business transformation and success. They will also be critical to building the trust necessary to expand adoption and to protecting the organization from regulatory and unintended negative consequences from these emerging approaches.