Magic Quadrant for Continuous Controls Monitoring
 
23 March 2010

French Caldwell, Paul E. Proctor

Gartner RAS Core Research Note G00174594
 

Continuous controls monitoring is an emerging governance, risk and compliance technology that monitors controls in ERP and other financial applications to improve financial governance, monitor and verify access and transactional rules, and automate audit processes.





What You Need to Know



Within the governance, risk and compliance (GRC) marketplace, continuous controls monitoring (CCM) is a set of technologies that assist the business in reducing business losses from fraud or failure to follow rules governing financial transactions, and improving performance through continuous monitoring (CM) and reducing the cost of auditing through continuous audit (CA) of the automated controls in ERP systems or other financial applications. CCM contributes value to risk management and compliance initiatives in three ways:

  • Lowering compliance costs — A CCM solution can reduce the cost of audits by eliminating much manual sampling and minimizing the time it takes to gather documentation.
  • Improving financial governance — CCM can increase the reliability of transactional controls, improve auditor trust and increase the effectiveness of antifraud controls.
  • Improving operational performance — CCM controls, such as those that monitor duplicate payments, incorrect discounts or misapplied warranties, go beyond what most people consider compliance. By preventing these violations of business rules, CCM can improve key financial processes and increase the availability of working capital.

The CCM market is in its early stages, and many vendors don't offer a complete solution, or they offer solutions that are targeted at specific ERP applications but don't work as well in heterogeneous financial application environments.

CCM is simplest and least expensive to apply in a homogeneous ERP environment for which the vendor has a preconfigured integration link and controls library.

When financial processes are spread across multiple instances, and especially when there is a mix of ERP financial applications and/or other non-ERP financial applications, customizing controls and application integration adds expense. Costs of customization should be balanced against what it would cost to migrate to a common ERP. CCM in combination with other integration technologies, such as business process management systems, which can automate the data collection across multiple platforms and reporting, could mitigate the need to move to a common ERP.

Because CCM can be an expensive proposition requiring considerable implementation services, to get full value, implementation should focus not only on compliance and audit needs, but also on business performance benefits and improving financial governance.






Magic Quadrant



CCM technologies are applied automatically and periodically to support the monitoring of controls of financial processes that are repeatable, consistent and predictable. CCM solutions support CM for financial managers and CA for internal auditors:

  • CM is a business management monitoring function used to ensure that controls operate as designed and that transactions are processed appropriately. CM uses control automation to reduce fraud and improve financial governance, typically resulting in a substantial return on investment. It improves the reliability of the controls, and it improves the management oversight, policy enforcement and operational efficiency for critical financial processes, often producing hard-dollar savings.
  • CA is the periodic collection of audit evidence and indicators for the benefit of internal audits. CA reduces audit costs by automating the audit process and eliminating the cost of manual sampling. To avoid audit deficiencies, it is important that policies are being followed demonstrably, and that exceptions are documented and proved to be within the boundaries of good practice.

There are four technologies that make up CCM:

  • CCM for segregation of duties (CCM-SOD) is used to manage a number of access conflicts present in ERP and financial applications. This Magic Quadrant replaces the "MarketScope for Segregation of Duty Controls Within ERP and Financial Applications."
  • CCM for transactions (CCM-T) is used to continuously monitor ERP and financial application transaction information to improve governance and automate audit processes.
  • CCM for master data (CCM-MD) automates controls related to ERP and financial application data. It is an element of many data quality products.
  • CCM for application configuration (CCM-AC) is used to monitor the presence, appropriate configuration and modification of built-in application controls. CCM-AC is used in conjunction with each of the other three CCM technologies.

Some CCM vendors offer audit analytics as part of a CCM solution set, and it can be an important feature for internal auditors. Audit analytics is similar to CCM, except that the queries are initiated manually, and not automatically. It is used by auditors during the course of periodic audits to run standard or ad hoc queries against sets of transactional data. It is also used to test for SOD, and to verify that accounting rules governing three-way match requirements, duplicate payments and various two-way match requirements are being met.

In the evaluation of vendors for this Magic Quadrant, the focus was on CCM-SOD and CCM-T. CCM-MD and CCM-AC are secondary controls technologies, in that SOD and transactional controls have master data and application configuration elements. For instance, to monitor for duplicate payments, which is a transactional control, master data monitoring would be needed to ensure that payment data is formatted in a comparable way and that it has not been changed. Also, application configuration monitoring would be important to ensure that the transactional controls within the payment process had not been changed.

Figure 1. Magic Quadrant for Continuous Controls Monitoring

Figure 1.Magic Quadrant for Continuous Controls Monitoring

Source: Gartner (March 2010)
 



Market Overview

The CCM market is relatively small and immature. Market penetration of CCM-SOD is about 25%, and CCM-T is less than 10%. However, despite the recession, the CCM market has grown at double-digit rates during the past year, with some of the smaller best-of-breed vendors reporting a doubling of software license revenue. The larger players in the market have seen smaller growth rates, ranging from 10% to 30% year over year.

With CCM's genesis in responding to Sarbanes-Oxley Act (SOX) compliance deficiencies, the market is largest in North America. However, in Europe, there have been some large implementations. CCM markets are developing in other geographies as well. Based on reports from CCM vendors, the market is roughly 70% North America, 20% Europe and 10% elsewhere, with Australia and Brazil being targeted as emerging markets for many vendors.

Regarding industry trends, CCM-SOD sales have not had a particular vertical focus. For CCM-T, banks have been the heaviest buyers, with the objectives of raising auditor productivity for regulatory compliance, better fraud monitoring and transactional process improvement. During the past year, other industries have shown interest in CCM-T. Notably, some large municipalities have issued RFPs for CCM, with the objective of raising auditor productivity.

There is a legal challenge before the U.S. Supreme Court regarding the constitutionality of one of the SOX regulators. Depending on the court's decision, the CCM and other GRC markets could see a decrease in the rate of growth. However, SOX is just one driver for this market. Starting in late 2008, enterprises began looking at the financial benefits of expanding their CCM from SOD to include transactional controls. The primary market drivers for CCM are as follows:

  • Regulatory compliance — Corporate governance and anti-corruption rules worldwide, which focus on corporate fraud, have heavily influenced the adoption of CCM. To comply with rules focused on corporate fraud, such as SOX and the Foreign Corrupt Practices Act (FCPA) in the U.S., SOD in financial applications has been a focus of auditors and regulators. When SOD became a common audit deficiency, particularly for SOX Section 404 audits, many enterprises looked to CCM-SOD to automate the monitoring of SOD controls within ERP and other financial applications. Until recently, SOX auditors have not put as much focus on transactional controls monitoring as they did on SOD. However, high-profile enforcement of anti-corruption regulations, such as FCPA in the U.S. and Gesetz zur Bekämpfung der Korruption in Germany, have shifted attention to CCM-T.
  • Risk management — Many enterprises still look at compliance as a bottom-up documentation and audit exercise to demonstrate adherence to individual mandates. However, in 2010, most have turned to risk management as a way to focus compliance resources on controls objectives that are at the highest risk. Once they make this transition, executives at these enterprises start to demand more business value from GRC investments. Controls automation to improve controls monitoring and audit is a way to provide that value. Boards also are directing improvements in enterprise risk management, including ongoing monitoring of risks, and not just annual assessments.
  • Business performance — CCM is part of the financial governance marketplace, as well as GRC. When applied to improving financial governance, not only can CCM help to ensure that policies are followed for compliance reasons, but also it has direct business performance benefits — particularly in reducing fraud and improving transactional processes — thereby increasing the availability of working capital.



Market Definition/Description

CCM is a set of technologies applied to the controls in financial applications to assist the business in reducing the cost of auditing through CA, reducing business losses through CM, and improving business performance by ensuring that automated process controls are working effectively. CCM technologies are applied automatically and periodically to support processes that are repeatable, consistent and predictable.

CCM software analyzes data in ERP and other financial applications to identify exceptions to policies, business rules and built-in application controls. CCM software can also be used to establish controls and monitor them. CCM software has several functions, including controls monitoring, exception and remediation management, reporting and analytics, and workflow:

  • Controls monitoring functions automatically and periodically import data from ERP and financial applications, and apply a set of predefined audit analytics to identify control exceptions.
  • Exception and remediation management supports tracking the response to identified control failures and other deficiencies, along with the process of addressing exceptions.
  • Reporting and analytics support trending and audit analysis, audit trails, dashboards and the generation of reports.
  • Workflow supports notifications, alerts, reviews, approvals and other process automation needs.

CCM technologies fall within the GRC marketplace, and a few vendors offer both CCM and enterprise GRC platform products. CCM is a subset of a broader set of technologies called "controls automation and monitoring", which include infrastructure, systems and other application controls. They have also been referenced as "controls-monitoring analytic applications" in the broader packaged financial application market.




Inclusion and Exclusion Criteria

To be included in this Magic Quadrant, vendors had to demonstrate functionality for either SOD or transactional controls monitoring (CCM-SOD or CCM-T, as described above). The controls monitoring must be done periodically and automatically.

Vendors with less than $2 million revenue in 2009 or with fewer than 15 customer implementations were excluded.

Some vendors were considered but not included in this Magic Quadrant for the following reasons:

  • AlertEnterprise did not meet production customers' thresholds. It has a unique value proposition for enterprises needing to combine the monitoring of physical and logical access controls for SOD. Besides the typical financial controls monitoring, AlertEnterprise can help with controls monitoring related to critical infrastructure protection.
  • BWise did not meet production customers' thresholds. It launched a CCM product in late 2009, which is integrated with its enterprise GRC platform offering.
  • CSI has an audit analytics solution for ad hoc queries that can be automated for CCM, but usually is not. It can be used to facilitate the rule definition for CCM solutions.
  • Lumigent automates the analysis of application database recovery log files that can provide additional assurance of master data integrity. It has a prebuilt, application-specific business rule layer for PeopleSoft, Lawson and Deltek that maps predefined application controls and policies to specific SOX reporting requirements, which reduces the amount of manual audit sampling needed. Lumigent has no direct access to transactional data for CCM-SOD and CCM-T.
  • SenSage did not meet production customers' thresholds. It has CCM-T and user activity monitoring for SAP, but it will not compete directly against SAP. SenSage partners with SAP to extend SAP GRC into security and activity monitoring for SAP, log management, and security information and event management.



Evaluation Criteria

Ability to Execute

Product — Core goods offered by the technology providers that compete in/serve the defined market. CCM vendors were evaluated for their ability to deliver CCM-SOD and CCM-T. Vendors that could deliver both scored better than those with just one of the capabilities. Further points were awarded for out-of-the-box integrations and control libraries for multiple ERP systems.

Overall Viability — Viability includes an assessment of the overall organization's financial health, the financial and practical success of the business unit, and the likelihood of the individual business unit to continue to invest in the product, continue offering the product and advancing the state of the art within the organization's portfolio of products. Revenue and revenue growth were the primary factors. For large ERP vendors, a discount factor was used to account for the fact that CCM represents a small percentage of overall revenue.

Sales Execution/Pricing — Each technology provider's overall effectiveness in sales was measured by sales growth in 2009, the size of the sales force and the extent of channel sales. Pricing models were also evaluated for transparency and ease of comparison to competitors.

Market Responsiveness and Track Record — Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve and market dynamics change. Because this market is in its early stages, the primary metric was overall customer implementations.

Customer Experience — Vendors provided reference customers. The references were evaluated not just for their satisfaction with the vendor and product, but also for the value of the reference itself: Large, complex projects with both CCM-SOD and CCM-T were more valuable and given more points than those with simple implementations. Full production examples were given more points than pilot implementations.


Table 1. Ability to Execute Evaluation Criteria

Evaluation Criteria
Weighting
Product/Service
Standard
Overall Viability (Business Unit, Financial, Strategy, Organization)
High
Sales Execution/Pricing
Standard
Market Responsiveness and Track Record
High
Marketing Execution
No rating
Customer Experience
Low
Operations
No rating

Source: Gartner (March 2010)

 



Completeness of Vision

Market Understanding — Ability of the technology provider to understand buyers' needs and translate those needs into products and services. Vendors that show the highest degree of vision listen to and understand buyers' wants and needs, and can shape or enhance those wants with their added vision. In the CCM market, vendors were scored against three market elements: audit, compliance and business performance.

Product Strategy — A technology provider's approach to product development and delivery that emphasizes differentiation, functionality, methodology and feature set as they map to current and future requirements. Four elements were considered: integration and support for multiple ERP systems, automation and workflow, reporting and analytics, and templates and modules.

Vertical/Industry Strategy — The technology provider's strategy to direct resources, skills and offerings to meet the specific needs of individual market segments, including industry verticals. Vendors were evaluated for their ability to support regulatory compliance and business performance strategies for multiple industry verticals.

Innovation — Direct, related, complementary and synergistic layouts of resources, expertise or capital for investment, consolidation, defensive or pre-emptive purposes. The fraction of revenue dedicated to R&D was a primary metric. Aggressiveness of the vendor's road map and a history of executing effectively against the road map also were considered.

Geographic Strategy — The technology provider's strategy to direct resources, skills and offerings to meet the specific needs of geographies outside the "home" or native geography, either directly or through partners, channels and subsidiaries, as appropriate for that geography and market. Sales, proven resellers and support in multiple geographies scored better than if the strategy was dedicated to just one or two geographies.


Table 2. Completeness of Vision Evaluation Criteria

Evaluation Criteria
Weighting
Market Understanding
High
Marketing Strategy
No rating
Sales Strategy
No rating
Offering (Product) Strategy
High
Business Model
No rating
Vertical/Industry Strategy
Low
Innovation
Standard
Geographic Strategy
Low

Source: Gartner (March 2010)

 



Leaders

The CCM market is far from mature, but the leaders in the market have had a significant presence in the market for many years. They all have strong, market-tested CCM-SOD capabilities and offer CCM-T. Their CCM-AC and CCM-MD capabilities are adequate to support primary CCM-SOD and CCM-T functions. When lacking in multiplatform capabilities, they have services and technology partnerships to fill the gap.




Challengers

Like the leaders, challengers have proved their ability to execute over several years. Challengers have the ability to deliver only CCM-SOD or CCM-T, or when they deliver both, they are strong in one and immature in the other. Their CCM-AC and CCM-MD capabilities are adequate to support primary CCM-SOD and/or CCM-T functions. When lacking in multiplatform capabilities, they have services and technology partnerships to fill the gap.




Visionaries

Visionaries have proven capability to deliver both CCM-SOD and CCM-T, and offer significant differentiation from competitors on their product, market or geographic strategies. Their financial viability may be lower, they may be slower to gain market acceptance, or their products may not be as mature as those of leaders.




Niche Players

Niche players exceed the minimum criteria to be in this Magic Quadrant, and have competitive offerings. Some niche players do not have a large number of dedicated ERP integrations out of the box, or they focus on a specific vertical market, or both. Other niche players have the ability to deliver only CCM-SOD or only CCM-T, or if they deliver both, they may not have proven yet that they have effective strategies that will differentiate them from competitors and grow their presence in the market.




Vendor Strengths and Cautions

ACL Services

Strengths
  • Product — For CCM-T, ACL Services offers very good out-of-the-box direct integration with SAP ERP financial management software.
  • Pricing — ACL offers a low-cost option for CCM-T, and its pricing model is easy to understand.
  • Market Responsiveness — ACL has a large customer base using its audit analytics product, which has provided a good foundation for adoption of its CCM offering.
  • Customer Experience — ACL has good references, illustrating a range of objectives from auditor productivity to improving business performance.



Cautions
  • Product — ACL has limited SOD, limited controls libraries, and limited out-of-the-box connectors for ERP other than SAP. Rather than SQL, ACL uses a proprietary scripting language — training in its use does not take long.
  • Product Strategy — Having provided services for many years to the audit community, ACL has a lot of experience and intellectual property that can be applied through service engagements. It's not clear how ACL will take this intellectual property and use it to improve its products.
  • Viability — ACL's slow growth is partly due to the price of its product and an older business model that is dependent on service revenue.



Approva

Strengths
  • Product — Approva has prebuilt integration links to multiple ERP vendors. It provides good workflow for exception management, robust reporting and intuitive rule building.
  • Viability — Approva is the largest best-of-breed vendor, and it has strong partnerships with Big 4 audit firms.
  • Geographic Strategy — It has customer implementations globally, with an emphasis on North America and Europe.



Cautions
  • Viability — Approva competes directly against megavendors Oracle and SAP. Besides CCM-SOD and CCM-T, Approva also offers an audit analytics capability that competes against ACL and CaseWare IDEA. Although Approva is the largest CCM best-of-breed vendor, it is still a relatively small software vendor, and it will be a challenge to maintain competition on multiple fronts.
  • Sales Execution — Approva's growth has slowed in the past year. In the past, it heavily focused on CCM-SOD, where the buyer was typically in the IT organization. As Approva increases its emphasis on CCM-T, it must increase its sales force's ability to reach finance and audit buyers.



Greenlight Technologies

Strengths
  • Product — Greenlight Technologies offers CCM-SOD and CCM-T, and it has preconfigured control libraries and connectors for multiple ERP systems. It also has intuitive navigation and workflow. It offers a wizard for users to automatically create controls without writing script.
  • Product Strategy — Greenlight provides connectors and cross-platform analysis capabilities that have made it a trusted partner of SAP. It has also partnered with Oracle to extend Oracle's cross-platform capabilities.
  • Customer Experience — It has good references for CCM-SOD and CCM-T.
  • Geographic Strategy — Greenlight has a strong partnership with SAP, and its development of channel partnerships with consulting firms gives Greenlight a large geographic reach for a small vendor.



Cautions
  • Innovation — Its R&D investment is relatively small.
  • Sales Execution — Greenlight is very dependent to date on its partnership with SAP for sales. It is building an independent sales capability.
  • Viability — Greenlight is small from a revenue standpoint, but it has good growth. Its growth has been dependent on being an SAP technology and implementation partner.



Infogix

Strengths
  • Customer Experience — Reference customers cited complex multiapplication environments and satisfaction with Infogix and its products. Infogix deploys products rapidly, and training time to learn how to build controls is short.
  • Innovation — Infogix invests significantly in product development, and it has a relatively large development workforce for CCM.
  • Product — It provides CCM-T for monitoring high-volume, multistep transactional processes. Infogix's products are used in environments where there are multiple heterogeneous financial systems, with transactional processes that cross those multiple systems, and that require building customized controls.



Cautions
  • Market Understanding — The CCM market is heavily focused on customers with ERP implementations, which is a market that is untested for Infogix because it focuses on specialized transaction systems.
  • Product — Infogix has no out-of-the-box integrations or controls libraries for ERP applications. It has no CCM-SOD.
  • Market Responsiveness — For CCM, Infogix's adoption is mostly customers of its other transactional control products.



Oracle

Strengths
  • Product — Oracle offers a complete CCM solution set for Oracle E-Business Suite and PeopleSoft Financials. It provides integration for compliance, risk management and business performance with Oracle's Enterprise GRC Manager and GRC Fusion Intelligence.
  • Product Strategy — Oracle is extending integrations to other Oracle transactional products, including Hyperion Financial Manager, as part of an overall Oracle business performance and financial governance strategy, as well as a strong GRC strategy.
  • Innovation — CCM is a growth area for Oracle, and it has increased its R&D investment in it.
  • Geographic Strategy — Oracle offers global marketing, sales and support.



Cautions
  • Product Strategy — Oracle has been slow to develop direct integrations with some of its own products, such as JD Edwards and competitors' ERP systems.
  • Market Understanding — Oracle is not aligned with the CA market. It is well-focused on CM for compliance and business performance.



Oversight Systems

Strengths
  • Product Strategy — Oversight Systems is focused on CCM-T in complex heterogeneous application environments and multivariable analytics, and on large-volume transactions.
  • Market Understanding — Its focus on audit and business performance is excellent.
  • Market Responsiveness — Oversight Systems has experienced high growth in direct competition with vendors that have stronger presence and name recognition in the analytics and GRC markets.



Cautions
  • Product — Oversight Systems' products have limited SOD. Navigation and workflow are sometimes not intuitive.
  • Pricing — Pricing is opaque, inconsistent and difficult to compare to models using user or module pricing models.
  • Innovation — Oversight Systems' R&D investment is relatively low.



SAP

Strengths
  • Product — SAP supports CCM-SOD and CCM-T for SAP ERP versions 4.6c and above out of the box. Integration with other versions of SAP and competitors' products is available with the standard license. Reporting has been significantly improved with SAP Business Objects integration.
  • Innovation — SAP is making a substantial investment in this growth area of its overall business strategy.
  • Sales Execution — SAP has executed extraordinarily well, particularly with CCM-SOD, with more than 2,000 customer implementations.
  • Geographic Strategy — SAP provides global marketing, sales and support.



Cautions
  • Market Understanding — In the past year, SAP has worked to develop a better approach to the market. In the past, it has marketed a number of products, including CCM, under the GRC banner, without clear differentiation as to the problems each solved for buyers.
  • Product Strategy — CCM-T is not differentiated well from SAP's compliance management offering, which causes market confusion. SAP plans to release a major upgrade to its GRC product set by the end of 2010, which, besides offering improved workflow and integration with other GRC products, should alleviate some of the product-positioning confusion.



Security Weaver

Strengths
  • Product — Security Weaver provides good support for CCM-SOD and CCM-T in SAP. It supports Oracle integrations out of the box for CCM-SOD, but not yet for CCM-T.
  • Customer Experience — Security Weaver presented one of the largest, most-complex and demanding CCM projects to date with very good results.
  • Innovation — Security Weaver is very responsive to customer recommendations and works closely with its customer base on development.
  • Viability — A lawsuit challenging its intellectual property rights was settled in December 2009, which has cleared concerns about Security Weaver's financial viability.



Cautions
  • Product Strategy — Security Weaver has had an SAP-centric focus in the past, but it has an aggressive release schedule for 2010 that will further extend capabilities for Oracle and other ERP systems, and improve workflow and reporting. Security Weaver has a proven track record on past releases.
  • Innovation — Security Weaver depends heavily on working with customers to develop new functionality, and independent R&D investment is low.
  • Market Responsiveness — Security Weaver experienced below-average growth in 2009, partially attributable to a lawsuit challenging its intellectual property rights.



SymSure

Strengths
  • Product — SymSure is oriented toward CCM-T in heterogeneous environments. It offers a generic controls library, and integrates with CaseWare IDEA and ACL for audit analytics capabilities.
  • Pricing — SymSure is a low-cost option for CCM-T.
  • Innovation — More than 60% of SymSure's revenue goes to R&D, which is a higher percentage than any other competitor.



Cautions
  • Product — SymSure doesn't have SOD, and there are no out-of-the-box integrations for ERP.
  • Viability — It is a small vendor.
  • Geographic Strategy — SymSure has sales and support offices only in North America and the Caribbean; however, it also has customers in the U.K., Latin America and Australia.

© 2010 Gartner, Inc. and/or its affiliates. All rights reserved. Gartner is a registered trademark of Gartner, Inc. or its affiliates. Reproduction and distribution of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner's research may discuss legal issues related to the information technology business, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The opinions expressed herein are subject to change without notice.






Note 1
Controls Automation and Monitoring




Controls automation and monitoring (CAM) is a broader category than CCM and includes many of the technologies that appear on the "Hype Cycle for Governance, Risk and Compliance Technologies, 2009." CAM includes any tools that act proactively to implement controls through business rules or reactively to monitor controls through the analysis of processes, transactions and events. Controls automation and monitoring can take many forms and operate at several levels of the enterprise architecture. At the infrastructure level, controls automation and monitoring focus on configuration management and network access. At the system level, they focus on identification and access. At the application level, they focus on SOD and, most recently, on rules governing transactions and behavior.





Vendors Added or Dropped




We review and adjust our inclusion criteria for Magic Quadrants and MarketScopes as markets change. As a result of these adjustments, the mix of vendors in any Magic Quadrant or MarketScope may change over time. A vendor appearing in a Magic Quadrant or MarketScope one year and not the next does not necessarily indicate that we have changed our opinion of that vendor. This may be a reflection of a change in the market and, therefore, changed evaluation criteria, or a change of focus by a vendor.





Evaluation Criteria Definitions





Ability to Execute

Product/Service: Core goods and services offered by the vendor that compete in/serve the defined market. This includes current product/service capabilities, quality, feature sets and skills, whether offered natively or through OEM agreements/partnerships as defined in the market definition and detailed in the subcriteria.

Overall Viability (Business Unit, Financial, Strategy, Organization): Viability includes an assessment of the overall organization's financial health, the financial and practical success of the business unit, and the likelihood that the individual business unit will continue investing in the product, will continue offering the product and will advance the state of the art within the organization's portfolio of products.

Sales Execution/Pricing: The vendor's capabilities in all presales activities and the structure that supports them. This includes deal management, pricing and negotiation, presales support and the overall effectiveness of the sales channel.

Market Responsiveness and Track Record: Ability to respond, change direction, be flexible and achieve competitive success as opportunities develop, competitors act, customer needs evolve and market dynamics change. This criterion also considers the vendor's history of responsiveness.

Marketing Execution: The clarity, quality, creativity and efficacy of programs designed to deliver the organization's message to influence the market, promote the brand and business, increase awareness of the products, and establish a positive identification with the product/brand and organization in the minds of buyers. This "mind share" can be driven by a combination of publicity, promotional initiatives, thought leadership, word-of-mouth and sales activities.

Customer Experience: Relationships, products and services/programs that enable clients to be successful with the products evaluated. Specifically, this includes the ways customers receive technical support or account support. This can also include ancillary tools, customer support programs (and the quality thereof), availability of user groups, service-level agreements and so on.

Operations: The ability of the organization to meet its goals and commitments. Factors include the quality of the organizational structure, including skills, experiences, programs, systems and other vehicles that enable the organization to operate effectively and efficiently on an ongoing basis.


Completeness of Vision

Market Understanding: Ability of the vendor to understand buyers' wants and needs, and to translate those into products and services. Vendors that show the highest degree of vision listen to and understand buyers' wants and needs, and can shape or enhance those with their added vision.

Marketing Strategy: A clear, differentiated set of messages consistently communicated throughout the organization and externalized through the website, advertising, customer programs and positioning statements.

Sales Strategy: The strategy for selling products that uses the appropriate network of direct and indirect sales, marketing, service and communication affiliates that extend the scope and depth of market reach, skills, expertise, technologies, services and the customer base.

Offering (Product) Strategy: The vendor's approach to product development and delivery that emphasizes differentiation, functionality, methodology and feature sets as they map to current and future requirements.

Business Model: The soundness and logic of the vendor's underlying business proposition.

Vertical/Industry Strategy: The vendor's strategy to direct resources, skills and offerings to meet the specific needs of individual market segments, including vertical markets.

Innovation: Direct, related, complementary and synergistic layouts of resources, expertise or capital for investment, consolidation, defensive or pre-emptive purposes.

Geographic Strategy: The vendor's strategy to direct resources, skills and offerings to meet the specific needs of geographies outside the "home" or native geography, either directly or through partners, channels and subsidiaries as appropriate for that geography and market.