How Products and Services Are Evaluated in Gartner Critical Capabilities

Overview

Analysis

Assessing the range of technology products or services in a given market is a crucial business task for an IT or buying department. Every range of products and services varies in terms of core features, target market, compatibility, expandability, user experience and other areas.

Gartner Critical Capabilities research delivers focused insight into providers’ product or service offerings and how they rate against common client-usage scenarios. This essential companion to the Gartner Magic Quadrant helps you determine which technology or service best fits your needs by providing a comparative analysis that scores competing products or services against a set of critical differentiators identified by Gartner. Thus, Critical Capabilities research shows, in graphical and tabular form, which products or services best fit the most significant usage scenarios for that market. This content will also help you assess the comparative strengths of multiple products and services against your current (and future) needs.

Use this research to investigate a range of product and service capabilities against several usage scenarios to help create a vendor shortlist for further evaluation.

Gartner’s Magic Quadrant and Critical Capabilities research are closely linked. The Magic Quadrant provides a broader view of the vendors in a particular market, while the accompanying Critical Capabilities analysis gives a more detailed look at the specific products or services from key vendors.

Magic Quadrants provide a holistic view of the set of vendor characteristics across 15 distinct criteria, focusing on vision and execution capabilities. Magic Quadrants go beyond the specifics of product or service offerings. Critical Capabilities focus on a group of competing products or services based on a set of use cases that match typical client deployment scenarios. These use cases are based on the real-world problems that clients need to solve, as well as how they intend to use the technology or service in their enterprises.

The correlation between vendor positioning in a Magic Quadrant and its product or service ratings in the Critical Capabilities is limited. Only one of the 15 possible Magic Quadrant assessment criteria — Product/Service under Ability to Execute — is closely connected.

A highly rated vendor in a Magic Quadrant can have a low-ranked product or service in the Critical Capabilities, because even leading vendors can produce products or services that miss the critical needs of the market. Conversely, lower-rated vendors in Magic Quadrants may struggle to make their businesses thrive, but produce a tightly focused and highly targeted product or service that outranks all other vendor offerings. Obviously, the general trend is for leading vendors to provide leading products and services, and the purpose of the Critical Capabilities is to provide an assessment framework for rating them (see Figure 1).

Like Magic Quadrants, Critical Capabilities research is an effective tool to help you better understand vendors and their offerings. Equally important, you should consult with a Gartner analyst who specializes in the market to gain details and insight affecting your specific needs and mission-critical priorities.

In defining a product or service category for evaluation, the analyst first identifies the leading usage scenarios for the products or services in this market.

The analyst answers the questions:

• What needs are end users looking to fulfill when considering products or services in this market?

• What usage scenarios are vendors typically targeting with their products or services?

The analyst models all of the common end-user deployment scenarios. The analyst will then combine some scenarios and eliminate other less significant ones to be left with between three and seven distinct end-user scenarios that define the use cases for the specific market.

Our methodology requires the analyst to then identify the “critical capabilities” for a class of products or services. These attributes differentiate products and services in a class in terms of their quality and performance. The analyst has to establish which capabilities form the most important criteria for acquisition decisions for the defined usage scenarios. The critical capabilities typically form a small subset or a grouping of the features commonly required by this class of products and services, not the entire range of capabilities that could be assessed. These “critical” capabilities should be distinct from the multitude of important criteria that would be common across all usage scenarios, but offer little differentiation. Some capabilities may be defined further to emphasize the differentiating factors of the capability and remove the assessment of important (but ever-present) baseline functionality. Each capability will be more or less important to every defined use case, which is represented by its weighting.

Critical Capabilities research is not intended to be an exhaustive analysis of every product, service and vendor in a market but rather, a focused overview. Vendors that appear in the Critical Capabilities report must have a product or service that could fulfill at least one of the differentiating capabilities.

Vendors that appear in the companion Magic Quadrant may or may not appear in the Critical Capabilities based on the inclusion criteria for the particular Critical Capabilities. Typically, a Magic Quadrant and its companion Critical Capabilities will share identical inclusion criteria, but this is not always the case.

Each vendor’s product or service is evaluated in terms of how well it delivers and differentiates on each capability, on a five-point scale. These ratings are displayed side by side for all vendors, allowing easy comparisons between the different sets of features.

Ratings and summary scores range from 1.00 to 5.00:

1 = Poor — Most or all defined requirements not achieved

2 = Fair — Some requirements not achieved

3 = Good — Generally meets requirements (may exceed some/not fully achieve others)

4 = Excellent — Meets or exceeds some requirements

5 = Outstanding — Significantly exceeds requirements

The analyst establishes the expected range for each capability scoring and what the characteristics of a 1 and a 5 would be. During evaluation, based on findings, an updated baseline for these expectations may be created to establish the final rating range.

Research activities include:

• Attending vendor briefings and product demonstrations

• Conducting surveys

• Interviewing reference customers identified by the vendors

• Interacting with industry contacts

• Discussing pertinent topics with clients and nonclients

• Consulting public sources, articles, speeches and published papers

• Gathering input from other Gartner analysts

• Assessing social media contributions, reviews and interactions, including Gartner Peer Insights

Teams of Gartner analysts collaborate to evaluate and rate each critical capability for each product or service against the rating criteria. The resulting product or service scores are used to generate a Critical Capabilities analysis.

To determine an overall score for each product in the use cases, each capability score is multiplied by the weighting factor for that capability and use case to come up with the use-case product score. The product scores for each vendor are independent and are not weighted or normalized again to produce the simple indexed ranking.

The Magic Quadrant and Critical Capabilities are typically published together (or in close proximity to each other), explaining each vendor’s position and rating, and comparing its products or services. As a result, these documents provide a context for making best use of any comparison. Before publication, the research undergoes rigorous internal peer review and validation, as well as a factual review by the vendors included in the Critical Capabilities.

Gartner is producing off-cycle updates to its Critical Capabilities research to reflect the latest material changes to a product or service’s capability score or fit for a use case in rapidly iterating markets. These updates highlight only major changes most relevant to technology portfolio decisions and include new analysis and adjusted scoring for affected vendors.

Analysts decide whether product/service changes materially affect the capabilities or fit for use case and update the named vendor scoring and profile if necessary. The annual Critical Capabilities process is unchanged. Updated capability scores and analysis feature in interactive and static Critical Capabilities. Gartner’s annual refresh cycle defines market scope, inclusion criteria, use cases, scoring norms and weightings.

The factors Gartner has selected do not represent all capabilities for a given product or service. Therefore, they might not represent the most important capabilities for a specific use situation or business objective. Gartner clients should use a Critical Capabilities analysis as one of several sources of input about a product or service before making a decision.

Gartner’s interactive Magic Quadrants and Critical Capabilities (which are available to entitled clients on gartner.com) enable you to gain additional insight into a market and its vendors, along with a deeper dive into the products and services they offer. You can use the interactive features to customize the use cases used by a Critical Capabilities to suit your business needs and requirements, and see which products or services fit best. You can also download all of Gartner’s data and use it as part of your procurement decision-making process.

The option to create client-customized Critical Capabilities content is intended to give you relevant insight into a market, product or service, but it does not represent Gartner’s view. Client-customized views of the Magic Quadrant and Critical Capabilities are for clients’ internal use only and are subject to our Gartner Usage Policy.

What is a Gartner Critical Capabilities analysis?

A Critical Capabilities report is an essential companion to the Magic Quadrant. It is a comparative analysis that scores competing products and services against a set of critical differentiators that are identified by Gartner. It shows clients which products or services fit best in various use cases and provides action-oriented advice to end-user clients on the products and services they should add to their vendor shortlists for further evaluation.

How does a Gartner Critical Capabilities differ from a Magic Quadrant?

A Critical Capabilities report is a companion note to a Magic Quadrant. The Magic Quadrant positions vendors in a market and presents a vendor’s broadly assessed performance relative to others in the same market. A Critical Capabilities report discretely analyzes the specific capabilities of products and services and presents the results against multiple usage scenarios.

What does it mean if a vendor isn’t included in a Critical Capabilities but appears in the companion Magic Quadrant?

Inclusion in the Critical Capabilities and Magic Quadrant reports are based on vendors meeting the inclusion criteria that are specific to each document. A Critical Capabilities may not feature all of the vendors in its related Magic Quadrant, but it could also feature additional vendors. Consult both documents for the differences in inclusion criteria. Differences in inclusion criteria that exclude a vendor from either report may not be relevant to your specific business need. You should not consider or eliminate a vendor from your selection process based on its inclusion (or otherwise) in both reports alone.

Are products and services analyzed in the same way every year?

No. As the market changes and client needs and requirements evolve, so does the analysis. Changes can occur in inclusion criteria, the use-case scenarios, the capabilities themselves and Gartner’s opinion on the features needed to meet the requirements.

How often are Critical Capabilities updated?

Critical Capabilities are typically updated annually, as are the Magic Quadrants. Gartner also publishes off-cycle event-driven updates to its Critical Capabilities to reflect material changes that significantly change a product or service’s capability score or fit for a use case. This may happen when a provider issues a new release of a product or service during the year, or as other significant events occur, such as the opening of new distribution channels. Our methodology ensures only material changes (i.e., those most relevant to technology portfolio decisions) drive the publishing of event-driven updates. Refer to Gartner Critical Capabilities: Event-Driven (Off-Cycle) Updates FAQs for more details.

My usage scenario doesn’t match any in the report; can I create my own?

Yes. Gartner clients with access to the interactive version of the Critical Capabilities research can modify an existing use case to create their own, based on their own capability weightings.

Can I add my own capabilities?

Adding new capabilities requires an assessment of each vendor product or service offering against each capability to attribute a rating against a set of predefined ranges. To support this effort, you can reuse the Gartner capability rating data to support your own internal assessment process. We recommend discussing any additional capabilities you want to add with a Gartner analyst to speed your process and avoid adding capabilities that may offer little differentiation.

What is the relationship between Magic Quadrants, Critical Capabilities and Gartner Peer Insights?

Gartner Peer Insights provides ratings and reviews of software and services contributed by IT professionals. Contributors share their firsthand experience of a vendor and product through the various stages of IT life cycle management, from evaluation through deployment. Gartner Peer Insights research uses the structure of the Market Description, Vendor List, Product List and Critical Capabilities (from the related Magic Quadrant and Critical Capabilities) for its user review survey structure and the client experience. Thanks to this innovative design (analyst-developed Magic Quadrants and Critical Capabilities, complemented by the perspective of Gartner Peer Insights), organizations can count on Gartner as their must-have source for a full-spectrum perspective on markets, vendors, products and services (see Explore Provider Offerings More Deeply With Gartner Critical Capabilities).

What if there is a difference between how a vendor or product is rated in Peer Insights and other Gartner research, such as Magic Quadrants and Critical Capabilities?

Peer Insights research reflects the individual opinions and experiences of end users who have submitted reviews and ratings through a rigorous verification, validation and moderation process. Analysts draw on a large base of information from multiple sources to contribute to their research. Although end-user feedback is important, so is feedback and input from vendors, public sources, investors, the press and social media. Lead authors of Magic Quadrants and Critical Capabilities will get access to Peer Insights review data that has been verified, validated and moderated. Analysts will be able to see data throughout the year to review crowd sentiment. A single Gartner Peer Insights review will not affect the placement of a vendor or product in the Magic Quadrant or Critical Capabilities. The Magic Quadrant and Critical Capabilities research process is based on a structured methodology to maintain independence and objectivity, and provide unbiased advice to clients.

Do Gartner’s research analysts use Peer Insights in their research?

Yes. Gartner Peer Insights reviews are one of the sources of customer input information that may be considered by Gartner experts as part of our rigorous research process. Gartner may consider other sources of customer input information in addition to Gartner Peer Insights, such as analysts’ inquiries with Gartner end-user clients and primary research survey results. Note that while end-user feedback is important, it is just one aspect in a vast range of criteria that are considered.

Gartner Recommended Reading