Assessing the range of technology products or services in a given market is a crucial business task for an IT or buying department. Every range of products and services varies in terms of core features, target market, compatibility, expandability, user experience and other areas.
Gartner Critical Capabilities research delivers focused insight into providers’ product or service offerings and how they rate against common client-usage scenarios. This essential companion to the Gartner Magic Quadrant enables you to determine which technology or service best fits your needs by providing a comparative analysis that scores competing products or services against a set of critical differentiators identified by Gartner. Thus, Critical Capabilities research shows, in graphical and tabular form, which products or services are a best fit in the most significant usage scenarios for that market. This content will also enable you to assess the comparative strengths of multiple products and services against your current (and future) needs.
Understanding Gartner’s Critical Capabilities Analysis
Gartner’s Magic Quadrant and Critical Capabilities research are closely linked. The Magic Quadrant provides a broader view of the vendors in a particular market, while the accompanying Critical Capabilities analysis gives a more detailed look at the specific products or services from key vendors.
Magic Quadrants provide a holistic view of the set of vendor characteristics across 15 distinct criteria, focusing on vision and execution capabilities. Magic Quadrants go beyond the specifics of product or service offerings. Critical Capabilities focus on a group of competing products or services based on a set of use cases that match typical client deployment scenarios. These use cases are based on the real-world problems that clients need to solve, as well as how they intend to use the technology or service in their enterprises.
The correlation between vendor positioning in a Magic Quadrant and its product or service ratings in the Critical Capabilities is limited. Only one of the 15 possible Magic Quadrant assessment criteria (Product/Service) on the Ability to Execute is closely connected.
It is possible that a highly rated vendor in a Magic Quadrant can have a low-ranked product or service in the Critical Capabilities, because even leading vendors can produce products or services that miss the critical needs of the market. Conversely, lower-rated vendors in Magic Quadrants may struggle to make their businesses thrive, but produce a tightly focused and highly targeted product or service that outranks all other vendor offerings. Obviously, the general trend is for leading vendors to provide leading products and services, and the purpose of the Critical Capabilities is to provide an assessment framework for rating them.
Like Magic Quadrants, Critical Capabilities research is an effective tool to help you better understand vendors and their offerings. However, it is equally important that you consult with a Gartner analyst who specializes in the market to gain details and insight affecting your specific needs and mission-critical priorities.
Defining Use Cases and Capabilities
In defining a product or service category for evaluation, the analyst first identifies the leading usage scenarios for the products or services in this market.
The analyst answers the questions:
The analyst models all of the common end-user deployment scenarios. The analyst will then combine some scenarios and eliminate other less significant ones to be left with between three and seven distinct end-user scenarios that define the use cases for the specific market.
Our methodology requires the analyst to then identify the “critical capabilities” for a class of products or services. These attributes differentiate products and services in a class in terms of their quality and performance. The analyst has to establish which capabilities form the most important criteria for acquisition decisions for the defined usage scenarios The critical capabilities typically form a small subset or a grouping of the features commonly required by this class of products and services, not the entire range of capabilities that could be assessed. These “critical” capabilities should be distinct from the multitude of important criteria that would be common across all usage scenarios, but that offer little differentiation. Some capabilities may be defined further to emphasize the differentiating factors of the capability and remove the assessment of important (but ever-present) baseline functionality. Each capability will be more or less important to every defined use case, which is represented by its weighting.
Critical Capabilities research is not intended to be an exhaustive analysis of every product, service and vendor in a market, but rather a focused overview. Vendors that appear in the Critical Capabilities report must have a product or service that could fulfill at least one of the differentiating capabilities.
Vendors that appear in the companion Magic Quadrant may or may not appear in the Critical Capabilities based on the inclusion criteria for the particular Critical Capabilities. Typically, a Magic Quadrant and its companion Critical Capabilities will share identical inclusion criteria, but this is not always the case.
Defining the Rating Criteria
Each vendor’s product or service is evaluated in terms of how well it delivers and differentiates on each capability, on a five-point scale. These ratings are displayed side by side for all vendors, allowing easy comparisons between the different sets of features.
Ratings and summary scores range from 1.00 to 5.00:
1 = Poor: Most or all defined requirements not achieved.
2 = Fair: Some requirements not achieved.
3 = Good: Generally meets requirements (may exceed some/not fully achieve others).
4 = Excellent: Meets or exceeds some requirements.
5 = Outstanding: Significantly exceeds requirements.
The analyst establishes the expected range for each capability scoring and what the characteristics of a 1 and a 5 would be. During evaluation, based on findings, it may be necessary to create an updated baseline for these expectations to establish the final rating range.
Researching Products and Services
Research activities include:
Attending vendor briefings and product demonstrations
Interviewing reference customers identified by the vendors
Interacting with industry contacts
Discussing pertinent topics with clients and nonclients
Consulting public sources, articles, speeches and published papers
Gathering input from other Gartner analysts
Assessing social media contributions, reviews and interactions, including Gartner Peer Insights
Teams of Gartner analysts collaborate to evaluate and rate each critical capability for each product or service against the rating criteria. The resulting product or service scores are used to generate a Critical Capabilities analysis.
To determine an overall score for each product in the use cases, each capability score is multiplied by the weighting factor for that capability and use case to come up with the use-case product score. The product scores for each vendor are independent and are not weighted or normalized again to produce the simple indexed ranking.
Publishing the Research
The Magic Quadrant and Critical Capabilities are typically published together (or in close proximity to each other) explaining each vendor’s position and rating, and comparing its products or services. As a result, these documents provide a context for making best use of any comparison. Before publication, the research undergoes rigorous internal peer review and validation, as well as a factual review by the vendors included in the Critical Capabilities.
Producing Event-Driven Updates for Each Product or Service as Needed
Gartner is producing off-cycle updates to its Critical Capabilities research to evaluate new ways to reflect the latest material changes to a product or service’s capability score or fit for a use case in rapidly iterating markets. These updates are created for products or services, as needed and include new write-ups and adjusted scoring. They highlight only major changes most relevant to technology portfolio decisions.
Analysts decide whether product/service changes materially affect the capabilities or fit for use case and update the vendor scoring and profile if necessary. The annual Critical Capabilities process is unchanged. Updates publish as separate documents linked to the main Critical Capabilities analysis. Updated capability scores also feature in interactive Critical Capabilities. Gartner’s annual refresh cycle defines market scope, inclusion criteria, use cases, scoring norms and weightings. Click here for more details.
Using Critical Capabilities Research for Your Requirements
The factors Gartner has selected do not represent all capabilities for a given product or service. Therefore, they might not represent the most important capabilities for a specific use situation or business objective. Gartner clients should use a Critical Capabilities analysis as one of several sources of input about a product or service before making a decision.
Gartner’s interactive Magic Quadrants and Critical Capabilities (which are available to entitled clients on gartner.com) enable you to gain additional insight into a market and its vendors, along with a deeper dive into the products and services they offer. You can use the interactive features to customize the use cases used by a Critical Capabilities to suit your business needs and requirements, and to see which products or services are the best fit. You can also download all of Gartner’s data and use it as part of your procurement decision-making process.
The option to create client-customized Critical Capabilities content is intended to give you relevant insight into a market, product or service, but does not represent Gartner’s view. Client-customized views of the Magic Quadrant and Critical Capabilities are for clients’ internal use only and are subject to our “Usage Guidelines for Gartner Services.”
Frequently Asked Questions
What is a Gartner Critical Capabilities analysis? ACritical Capabilities report is an essential companion to the Magic Quadrant. It is a comparative analysis that scores competing products and services against a set of critical differentiators that are identified by Gartner. It shows clients which products or services are a best fit in various use cases and provides action-oriented advice to end-user clients on the products and services they should add to their vendor shortlists for further evaluation.
How does a Gartner Critical Capabilities differ from a Magic Quadrant? A Critical Capabilities report is a companion note to a Magic Quadrant. The Magic Quadrant positions vendors in a market and presents a vendor’s broadly assessed performance relative to others in the same market. A Critical Capabilities report discretely analyzes the specific capabilities of products and services and presents the results against multiple usage scenarios.
What does it mean if a vendor isn’t included in a Critical Capabilities, but appears in the companion Magic Quadrant? Inclusion in the Critical Capabilities and Magic Quadrant reports are based on vendors meeting the inclusion criteria that are specific to each document. A Critical Capabilities may not feature all of the vendors in its related Magic Quadrant, but it could also feature additional vendors. Consult both documents for the differences in inclusion criteria. Differences in inclusion criteria that exclude a vendor from either report may not be relevant to your specific business need. You should not consider or eliminate a vendor from your selection process based on its inclusion (or otherwise) in both reports alone.
Are products and services analyzed in the same way every year? No. As the market changes and client needs and requirements evolve, so does the analysis. Changes can occur in inclusion criteria, the use-case scenarios, the capabilities themselves and Gartner’s opinion on the features needed to meet the requirements.
How often are Critical Capabilities updated? Critical Capabilities are typically updated on an annual basis, as are the Magic Quadrants. Gartner also publishes off-cycle event-driven updates to its Critical Capabilities to reflect material changes that significantly change a product or service’s capability score or fit for a use case. This may happen when providers issue a new release of a product or service during the year, or as other significant events occur, such as the opening of new distribution channels. Our methodology ensures that only material changes (i.e., those most relevant to technology portfolio decisions) drive the publishing of event-driven updates. Click “Gartner Critical Capabilities Event Driven Off Cycle Updates FAQs” for more details.
My usage scenario doesn’t match any in the report; can I create my own? Yes. Gartner clients with access to the interactive version of the Critical Capabilities research can modify an existing use case to create their own, based on their own capability weightings.
Can I add my own capabilities? Adding new capabilities requires an assessment of each vendor product or service offering against each capability to attribute a rating against a set of predefined ranges. To support this effort, you can reuse the Gartner capability rating data to support your own internal assessment process. We would recommend a discussion of any additional capabilities you want to add with a Gartner analyst to speed your process and avoid adding capabilities that may offer little differentiation.
What is the relationship between Magic Quadrants, Critical Capabilities and Gartner Peer Insights? Gartner Peer Insights provides ratings and reviews of software and services contributed by IT professionals. Contributors share their firsthand experience of a vendor and product through the various stages of IT life cycle management, from evaluation through deployment. Gartner Peer Insights research uses the structure of the Market Description, Vendor List, Product List and Critical Capabilities (from the related Magic Quadrant and Critical Capabilities) for its user review survey structure and the client experience. Thanks to this innovative design (analyst-developed Magic Quadrants and Critical Capabilities, complemented by the perspective of Gartner Peer Insights), organizations can count on Gartner as their must-have source for a full-spectrum perspective on markets, vendors, products and services (see “Explore Provider Offerings More Deeply With Gartner Critical Capabilities”).
What if there is a difference between how a vendor or product is rated in Peer Insights and other Gartner research, such as Magic Quadrants and Critical Capabilities? Peer Insights research reflects the individual opinions and experiences of end users who have submitted reviews and ratings through a rigorous verification, validation and moderation process. Analysts draw on a large base of information from multiple sources to contribute to their research. Although end-user feedback is important, so is feedback and input from vendors, references, public sources, investors, the press and social media. Lead authors of Magic Quadrants and Critical Capabilities will get access to Peer Insights review data that has been verified, validated and moderated. Analysts will be able to see data throughout the year to review crowd sentiment. A single Gartner Peer Insights review will not affect the placement of a vendor or product in the Magic Quadrant or Critical Capabilities. The Magic Quadrant and Critical Capabilities research process is based on a structured methodology to maintain independence and objectivity, and to provide unbiased advice to clients.