Care delivery organizations (CDOs) can gauge the risk of upgrading or implementing a healthcare application based on the number of sites that have already gone through the process. CIOs should be familiar with Gartner's new application risk model in order to set expectations and be prepared for possible problems.

• Implementing new healthcare applications and upgrading to newer versions of existing products can entail significant risk.
• Risk can be judged to a large degree by the number of sites that have successfully implemented the product.
• Type C organizations should be very wary of implementing a product with a risk level above 2.

• Have a good understanding of the maturity and associated operational risk level of products under consideration.
• Use the Gartner risk model to correctly set expectations, increase the probability of success and, by being aware of the risks, be in a better position to negotiate beneficial contracts.

CDOs are increasingly reliant on IT for automation and optimization of their mission-critical clinical and business processes. More organizations around the world are seeking to implement computer-based patient record systems. In the United States, the American Recovery and Reinvestment Act (ARRA), with its $19 billion specifically targeted for healthcare IT and "meaningful use" criteria, is prompting CDOs either to purchase a clinical system or to re-examine their existing clinical systems (see "Early Analysis of the Impact of the ARRA on U.S. Healthcare IT" and "Meaningful Use and Certification of EHRs: Tracking Evolving Targets"). At the same time, many revenue cycle management applications are nearing their life expectancy, and CDOs are beginning to consider their replacements. Furthermore, CDOs are seeking new categories of applications to improve patient access and throughput, as well as business intelligence applications to help improve operational and clinical efficiency and effectiveness.

As CDOs assess available products or consider upgrades, it is essential to correctly set internal expectations based in part on the possible difficulties that will occur during implementation. This will materially affect the probability for success of their new systems. As part of this evaluation, Gartner puts forth the following categories and associated criteria for judging the risk of a new product, based on the actual number of complete installations. For the purposes of evaluation, a product can be considered a full system (such as a financial system) or a module of a more-complete system (such as computer-based physician order entry [CPOE] as part of a computer-based patient record [CPR] system).

Vendors' products traverse a challenging path, from early development through immaturity to reliable, repeatable business value. "Maturity" does not just mean that the software itself is successfully deployed. It also means that surrounding elements are also fully developed and proven. These typically include user training and documentation, an implementation methodology with project plan/resource requirement templates for both vendor and client, vendor help desk/support systems and staffing, and processes for prioritizing enhancements, rapid prototyping/user review of new functionality, deploying new releases, and sharing user best practices. Product maturity materially affects the probability for success of a CDO's implementation of the system, as well as the relative time and resources required for that implementation. Correctly assessing, communicating and appropriately managing risk are critical functions of the CIO in partnership with the CDO's IT governance committee. This is an area of IT governance that healthcare CIOs often identify as a weak spot (see "A Benchmark of Healthcare IT Governance and Approaches for Improvement, 2008"). The actual number of full installations is a key metric to help assess the risk of a new product or a new version of an existing product.

Of course, certain applications can be riskier than others (for example, it can be very difficult to implement a CPR system with advanced clinical decision support and even more difficult to ensure that clinicians will use it). This model is generic enough to be applicable to all applications.

Overall, a new product undergoes distinct phases of evolution with respect to the code and to the implementation of the code. As a product matures, the code becomes more stable. Of course, if the vendor has superior software engineering, then the code should be stable once it becomes generally available, although all too often this is not the case. Perhaps even more important are the three phases that a vendor organization goes through with respect to a product. During the first few implementations, developers are critical for the installation, and the vendor's implementation team assists them. Then comes a period of time when the implementation teams are doing installations, but have very few standard experiences. Finally, as more sites are brought live, the implementation teams have done enough installations to have worked out a common methodology. It is this evolution that forms the crux of the new product risk model.

A vendor without a single implemented system or version must be considered to be in development. This phase can be further subdivided into three categories: conceptual phase, in development, and awaiting first implementations. A potential client must be very wary of a product that is in the conceptual phase (exists as a vision statement only). Purchasing such a product is heavily dependent on the expectation that the vendor can and will deliver the product on time and as expected. Although not a perfect guide, it is essential to understand the vendor's past performance in on-time delivery of useful new products, and plan accordingly. If the code is currently being written, then the vendor should be able to supply a reliable assessment of when delivery can be expected, but clients should still expect significant delays. Finally, the product may have been written, and the vendor may have identified alpha sites, but is still awaiting the first contract or the start of implementation. CDOs selecting a product in the development phase must be prepared for the virtual certainty of significant delays in delivery and installation. Contract negotiations should include vendor performance penalties for failure to deliver, as promised, in time and functionality. Only Type A (technically aggressive, well-funded organizations where IT is used to gain a competitive advantage) CDOs that also have a culture and history of successfully undertaking projects with undeveloped software should pursue Level 5 risk application projects.

If a vendor has fewer than four sites, the product must be considered to be in the beta testing phase. Note should be made that some vendors have been known to call their alpha sites "beta sites" and not identity their beta sites as such. Thus, the Gartner model is based on the number of implementations as opposed to the vendor's designation. In most cases, as the product is rolled out, there will be significant functionality issues and product defects. A client should receive some form of incentive or reduced license fees for being a beta site, because this generally implies the obligation to evaluate the product, help in defining and refining requirements, report needed changes back to the vendor, and host visits by prospective clients. Vendors will be highly invested in ensuring that the beta sites are successful and will devote a great deal of resources to the product to make up for the fact that implementation, training and support procedures, and documentation are incomplete. Oftentimes, developers make up a large proportion of the implementation team and will need to act as on-site troubleshooters. Clients must be prepared to devote a significant amount of time and resources to these implementations. Clients with beta systems should expect exemplary support, coupled with unpredictable delays and potential operational disruptions. Serious consideration must take place before deciding to become a beta site for mission-critical applications. Once again, Type A organizations are best suited for this type of project.

Even when a vendor has moved beyond the beta phase, CDOs should not expect that implementations will go smoothly or that the products will be without functional issues. Although most of the major code difficulties should have been identified by this phase, as more sites go live, other unanticipated problems may arise. As with the beta sites, vendors are still highly interested in demonstrating the value of their software and will devote more resources to implementations than they will use once the product and implementation methodology are mature. At this stage, training, support, and implementation procedures and documentation are still in flux — creating customer support and satisfaction issues. Type B CDOs (mainstream IT users with adequate funding that use IT for productivity) willing to take some chances might consider these products. Risk-averse CDOs must recognize that these products are not yet proven.

At this point, it is unlikely that major product issues would arise, as long as some of the implementations are in CDOs similar in size and case mix to a CDO that is considering the software. However, vendors are likely to find it difficult to devote the same number of resources to each new site as they did for Level 3 installations. In this phase, vendors are finalizing their standard implementation strategies, ramping up implementation teams and moving away from development team direct support. Application support should have been moved into the mainstream vendor customer support operations. Potential clients can be reasonably sure that the product seen at reference sites will be the product delivered, and should have a fairly accurate sense about what to expect during implementation.

Once a vendor has more than 20 sites that are fully implemented, it has enough experience and sufficient reference sites that it is unlikely that a potential client will experience major, unforeseen product or implementation difficulties. Training, implementation and support procedures should be stable and well documented. In this phase, the vendor may begin to train consulting companies to support implementation processes for clients. At this point, more of the risk of implementation lies with inadequacies in individual project planning or resource availability. Even at this stage, there may be surprises or challenges if a particular CDO's profile is dramatically different from the sites already implemented (for example, CDOs that are substantially larger in volume or complexity than previous implementations or that are attempting to roll out a CPR system in a children's hospital or behavioral health environment). Also, CDOs need to take into account that, once a product has reached the mature phase, it may have less functionality than some newer products, and that there might be increased pressure to move to that newer version sooner than if a less-mature product had been implemented.

When choosing to implement a new application or upgrade an existing one, CDOs need to carefully match their risk aversion, the criticality of the applications and the maturity of the product. The new product risk model provides CDOs with another set of data points on which they can make better decisions or at least set expectations correctly. When mapped against Gartner's Hype Cycle methodology, CDOs have a powerful tool to help mitigate their risk (see "Understanding Gartner's Hype Cycles, 2009").

Ensure that you understand how risk-averse the organization is (Type A, B or C). Does it fall on one side of the spectrum, seeking the latest in technology and being prepared for a number of substantial obstacles, or the other extreme, without any intent to implement anything until there are at least several sites identical to the first also running the software in question? The vast majority of CDOs are Type C and should wait for risk Level 1. Type B organizations willing to accept some risk can consider looking at risk Level 3 products. Only Type A organizations should consider risk Level 4 or 5 (see Figure 1).

Check the healthcare Hype Cycles (see "Hype Cycle for Healthcare Provider Technologies and Standards, 2009," "Hype Cycle for Healthcare Provider Applications and Systems, 2009" and "Hype Cycle for Telemedicine, 2009") to help determine whether or not the product appears ready.

Use the Gartner risk model to help set expectations for the success of a given implementation.

"Early Analysis of the Impact of the ARRA on U.S. Healthcare IT"

"Meaningful Use and Certification: Tracking Evolving Targets"

"Understanding Gartner's Hype Cycles, 2009"

"Hype Cycle for Telemedicine, 2009"

"Hype Cycle for Healthcare Provider Applications and Systems, 2009"

"Hype Cycle for Healthcare Provider Technologies and Standards, 2009"