1.0 What You Need to Know Every year, Gartner's analysts offer their predictions on what they see as the key issues facing the market spaces they cover. The storage research community's predictions for 2010 and beyond address changing hardware, software and service trends in the management of enterprise data. Among the trends we have identified in this research document are significant changes in storage hardware infrastructures, storage software management tools, storage services, and the kind and quantity of vendors that will deliver future storage products to private companies and to the cloud. 
2.0 Strategic Planning Assumptions Strategic Planning Assumption: By 2013, all 15,000 rpm HDDs and all FC-interface HDDs will be effectively extinct
- Although hard-disk drives (HDDs) have made enormous and critical advances in capacity and reliability, combined with incredible reductions in cost per gigabyte, the performance characteristics of HDD technology have not scaled in similar ways.
- With HDDs, enterprise input/output performance per gigabyte has actually declined in the past decade; this has created an “I/O bottleneck” that has grown at greater rates in recent years, which has opened up a greater need for the kind of ultrafast, low-latency access times provided by enterprise-grade (e-grade) SSDs.
- Correctly configured HDD and SSD combinations can help to unleash the full potential of multicore CPUs, increasing enterprise data throughput while decreasing footprint and power requirements.
Market Implications: Looking at the distribution of HDD interfaces in multiuser applications since 2002, Parallel Advanced Technology Attachment (PATA) and subsequently Serial ATA (SATA) dominated all entry-level server markets, and SATA will continue to dominate in low-end servers and near-line storage applications as well as in the new high-end external controller-based (ECB) systems that utilize e-grade SSDs. Small Computer System Interface (SCSI) and subsequently Serial Attached SCSI (SAS) HDDs have dominated and will continue to dominate in legacy direct-attached storage (DAS) markets and in midrange ECB storage systems. Fibre Channel (FC) HDDs have dominated in some midrange and most high-end ECB systems, but their share will decline. In dual-ported configurations, 2.5-inch small form factor (SFF) SAS HDDs should have a decreased power budget compared with 3.5-inch FC HDDs, and there will be no 2.5-inch SFF FC-interface HDDs; this will accelerate the decline of FC-interface HDDs beginning in 2010 as SFF gains share. FC-interface and all 15,000-rpm HDDs will disappear entirely during the 2009 through 2013 forecast period, partially because 4 Gbps FC will be the last of the HDD-based FC interfaces (whereas the faster 6 Gbps SAS HDD interface is already beginning to ship in volume) and also because e-grade SSDs will be increasingly used as the high-speed front end in Tier 0 or Tier 1 storage systems. But SSDs will only minimally displace HDDs, which will continue to have enormous cost and capacity advantages compared with e-grade SSDs. Even in 2013, Gartner believes that at the OEM component level, there will still be a 15x to 30x cost-per-gigabyte advantage of HDDs over SSDs ($0.03 to $0.06 compared with $1.00), while HDDs will have a 7x to 13x capacity advantage over e-grade SSDs (2GB to 4GB compared with 300GB). And even in 2013, there will continue to be a greater-than-$100 price premium at the OEM component level for choosing an e-grade SSD as opposed to an e-grade HDD. For these and other reasons — such as the assumption that the number of enterprise petabytes that need to be managed and stored will continue to expand at more than 40% per year, and the hard fact that the whole NAND flash industry can produce less than one-tenth the number of petabytes the HDD industry can produce — Gartner believes that shipments of multiuser SATA-interface HDDs will grow from 15.3 million units in 2009 to 21.2 million units in 2013; shipments of SAS-interface HDDs will grow from 17.5 million units in 2009 to 28.1 million units in 2013; shipments of FC-interface HDDs will decline from 7.1 million units in 2009 to an effective extinction by the end of 2013; and e-grade SSDs dedicated to storage (rather than to embedded applications in telecom, military or medical systems) will grow from 281,000 units in 2009 to 5.3 million units in 2013. At Gartner's December 2008 Data Center Conference, 97% of the IT professionals attending the enterprise SSD presentation indicated that they planned to integrate some form of e-grade SSD technology in their storage infrastructure within the next five years. But more than 80% of the attendees admitted that they did not clearly understand the costs and benefits of e-grade SSD technology. Notwithstanding the obvious need for better education and marketing, the critical “weight” of emerging e-grade SSD technology is such that no major storage company can afford not to offer compelling and competitive SSD solutions, which will become increasingly ubiquitous in storage infrastructures. However, in terms of the relative weight of data that will be managed and stored, we believe e-grade SSDs will be the "tip of the iceberg" — HDDs will power the massive bulk of storage content hidden in the "ocean" of available data.
- Today, Gartner is advising all clients involved with enterprise storage infrastructures to explore and evaluate (but not necessarily adopt) higher-end SSDs as potential storage network accelerators (SNAs).
- During the next 12 months, specific enterprise network and storage verticals may materially benefit from immediate implementations of SSD technology, but clients should expect it to come at a premium price. Gartner advises most users to watch and wait for enhanced SSD controller technology and new storage management software customized for SSDs.
- In 2011 and beyond, users should have increasing confidence in the enterprise SSD solutions offered by a variety of vendors. We are already seeing that large and small storage companies consider e-grade SSD offerings in some portion of their product portfolio to be a competitive necessity, not an option.
- OEMs and designers of storage systems should plan on the eventual demise of all 15,000 rpm and FC-interface HDDs and focus future architectures on 10,000 rpm, SFF, SAS- or SATA-interface HDDs; 3.5-inch, 5,400/5,900/7,200 rpm SAS- or SATA-interface HDDs; and SAS- or SATA-interface e-grade SSDs in diverse form factors.
"Emerging Technology Analysis: Solid-State Drives for Enterprise Storage" "Dataquest Insight: Evolving Technologies Enhance New Generations of PC-Grade SSDs" “Dataquest Insight: Global Hard-Disk Drive Forecast Scenarios, 2009-2013, 3Q09 Update” Strategic Planning Assumption: By 2013, object-based storage will be a key technology for managing unstructured data in the cloud Analysis by: Pushan Rinnen and Valdis Filks
- Object-based storage is transitioning from its first phase of content-addressable storage (CAS) to a second phase of cloud storage.
- Object-based storage can set automated storage policies and security policies at a more-granular level than traditional volumes and file systems, thereby meeting more closely cloud storage requirements.
- Object-based storage can scale into petabytes of capacity with lower cost and easier management than traditional SAN and NAS storage.
- The new-generation object-based storage is immature today but has great potential.
Market Implications: Object-based storage first emerged in the form of CAS in commercial systems that focused on disk-based archiving within a data center for regulatory compliance. Because of the lack of product availability from major vendors, limitations in product features and costs compared with tape, deployment of CAS systems has been mostly confined within a few vertical industries that have established stringent regulations requiring tamperproof disk storage. However, Web 2.0 in the past few years triggered a new wave of Web-based (or “cloud”-based) applications and businesses that require secure, low-cost storage systems with high-capacity scalability, more-intelligent global content distribution and policy-driven automation for simplified storage management. As a result, new commercial object-based storage solutions have emerged recently to target “cloud” applications such as global content distribution and low-cost cloud storage for backup and archiving. The new commercial solutions from storage vendors share similar characteristics. They all leverage the innate capability of object-based storage, which can associate each object with user-defined security and storage policies (such as creating multisite replicas). They are also built on a scale-out architecture using low-cost commodity servers with embedded storage. And they leverage standard Web protocols such as HTTP. We believe that the new-generation object-based storage from storage vendors is the first step toward viable commercial solutions for the cloud storage infrastructure. Although they will have limited near-term impact on major service providers in the public cloud, which may continue building their infrastructure leveraging their own in-house talent and open-source software, Gartner believes that they will have significant impact on private clouds and smaller service providers in the public cloud, which do have sufficient internal resources to build their own storage infrastructure. Nevertheless, some storage vendors may be reluctant to enter this market for fear of low margins. If product availability continues to be limited with little competition from major vendors, pricing for the new-generation object-based storage systems may not become low enough for the public cloud, and the impact of these systems may be confined within private clouds. But the private clouds alone still represent large lucrative opportunities that few vendors can afford to ignore.
- CIOs should start evaluating and plan to use the new-generation object-based storage to manage unstructured data in 2010 within their private cloud.
- Users should demand that object-based storage vendors use industry-standard protocols to minimize device and vendor lock-in and facilitate easier migration for long-term archiving.
- Storage vendors offering object-based storage should tie it to their existing data center ecosystem solutions.
“Cloud Storage: Benefits, Risks and Cost Considerations” Strategic Planning Assumption: By 2013, 50% of organizations with more than 1,000 employees will move 80% of their archived data to the cloud Analysis by: Sheila Childs and Adam Couture
- In 2008, the number of customers using service providers for e-mail archiving grew 59%, representing 42% of the overall e-mail archiving market; other content types will follow this trend.
- Organizational acceptance of data archiving in the cloud is most common among small to midsize enterprises today, but adoption of cloud archiving by large enterprises will occur beginning in 2010.
- Organizations will look to new vendors, rather than traditional on-premises software product providers, for cloud archiving solutions.
Market Implications: The business dynamics for moving archived data to a low-cost management model are clear. Organizations see archiving as a necessary function that varies little across their industries, is fairly simple, and does not offer a competitive business edge. In fact, archiving is far removed from an organization’s core business but must be implemented to manage data for compliance, risk management and records management. Consider the government organization or financial institution that has petabytes of older data: The difficulty and expense of managing this massive quantity of data, the minimal need for access to it, and the unwillingness to delete anything are causing organizations to look for cost-effective deployment models such as those that can be found in the cloud. Storage and archiving software vendors have responded to these evolving needs with new technologies, promoting platforms that are suitable for private and public clouds, and increasing their sales and marketing efforts to service providers. New solutions offer better floor space utilization and power efficiency. Multitenancy and unique namespace support for logical partitioning to accommodate multiple clients is becoming commonplace. Highly available distributed object stores for unstructured content can buttress archive-specific functionality, such as broad search capabilities, security, retention and immutability. A plethora of new cloud-based archiving service vendors has emerged, and with a few exceptions, they are separate and distinct from those offering traditional on-premises solutions. These new vendors have been and will continue to be attractive to small to midsize organizations. Large and well-known vendors — such as Microsoft, Google, Dell and Iron Mountain — have also begun to offer cloud archiving services, and as these vendors begin to prove their cloud archiving capabilities, enterprise adoption will increase, with larger organizations looking to these vendors for services. To ensure continued viability of their solutions, on-premises archiving software vendors will need to evolve their pricing models to be more competitive with hosted solutions. They will need to create additional capabilities to add value to on-premises solutions; for example, by positioning archive repositories as knowledge repositories, where archived data can be reused and repurposed for business value by being tightly integrated with ECM and other on-premises data management solutions. The dynamics of “buying archiving” will also change — buyers for archival storage and software become the service providers themselves rather than individual organizations, whose IT management will cease to care about archiving hardware and software but rather about data preservation and access services.
- Organizations must evaluate the cost models associated with on-premises archiving solutions versus cloud-based solutions in the context of their data preservation and access requirements.
- Enterprises must carefully evaluate a potential archive service provider’s solution to ensure that it can scale to meet their requirements and provide the specific services they need.
- Small to midsize organizations should look beyond traditional on-premises archive software and hardware vendors to well-established but not widely known cloud archiving service providers.
“Archiving in the Cloud: What End Users Should Consider” “Dataquest Insight: E-Mail Archiving Software Market, Worldwide, 2009” “Outsourcing E-Mail Archiving: 2Q09 Update” “IT Procurement Best Practice: Use a SaaS Pilot Phase” “Dataquest Insight: Evolving Market Dynamics for E-Mail Archiving SaaS” Strategic Planning Assumption: By 2013, the majority of primary NAS storage will be deduplicated, reducing capacity costs by 50% Analysis by: Stanley Zaffos, Valdis Filks and Pushan Rinnen
- Primary storage deduplication is predominantly available today in NAS and file-based products, and this trend will continue.
- Deduplication of primary NAS has the potential to reduce storage capacity costs by 33% to more than 85%, depending on recurring bit patterns within blocks of data and the size of the data pools. It has also proved to reduce storage capacity requirements by more than 50% in virtualized server environments when using NFS to support VMware ESX servers.
- Modern software-intensive storage appliances based on underlying file systems and powered by high-performance, multicore processors have no significant overhead associated with data deduplication.
- Primary NAS deduplication will improve performance for data services such as replication, snapshots and cloning, as well as backup and restore.
Market Implications: Backup and restore architectures, solutions and products will need to change to accommodate primary NAS deduplication, which has been driving NAS storage to faster growth rates than SAN storage. Deduplication within block-based primary storage products will continue to lag behind file-based NAS storage products by 18 to 24 months because of the performance priority of block-oriented applications and the difficulty of incorporating any form of data compression into external, control-block storage systems. Primary deduplication will drive the requirement for more processing power within storage devices. SSD or RAM storage will be used to store performance-critical, deduplication metadata structures and indexes. Current primary deduplication solutions can be obtained as nonchargeable inclusive options when purchasing storage devices. If this trend continues, then storage vendors may be forced to change their business models by including other data services within the base price. Primary NAS deduplication will become more efficient as more vendors offer deduplicated primary NAS and spend more research and development dollars for innovation, resulting in increasingly widespread adoption of the technology.
- Gartner is advising enterprise customers to quantify the potential savings of primary storage deduplication and operational impacts of changing the ratio of NAS to SAN storage when refreshing their infrastructure.
- System integrators should design backup and restore architectures to be flexible, so that they can adapt to already deduplicated primary storage data without the need to reduplicate and rehydrate and then redundantly deduplicate the same data by means of a hardware or software backup/restore deduplication product.
- To increase primary deduplication ratios and savings, users must create storage pools that contain common data used by different departments and applications.
- All major storage vendors must consider offering compelling, competitive primary deduplication solutions.
“Emerging Technology Analysis: Primary Data Deduplication, Storage Software and Hardware Technologies” “Best Practices for Turning a Storage Strategy Into Tactical Actions” “Best Practices for Lowering Your Storage Costs” “Use Data Deduplication to Improve Availability and Lower Cost” “EMC Acquires Data Domain, Becomes Deduplication Leader, Signals Deduplication as a "Must Have" Capability”
In response to your requests, we are taking a look back at a few key predictions from previous years. We have intentionally selected predictions from opposite ends of the scale — one where we were wholly or largely on target, as well as one we missed. On Target (2003): More and more users will demand that servers and storage systems take up less physical space and consume less power. The trend toward greater storage efficiencies, particularly with regard to footprint and power consumption, has become much more crucial — and obvious — in recent years, but we called this trend early and often, reiterating our stance in Gartner PlanetStorage conference keynotes and in multiple documents that reviewed new products, new technologies and new ways to integrate more cost-effective and productive storage infrastructures. New dimensions in corporate and personal computing have demanded and will continue to demand — among other things — more storage capacity and performance in less space requiring less power while providing seamless access to enormous databases; more degrees of data security and more widespread uses of data encryption; better archiving and disaster recovery systems; better data management and enterprise content management; better compliance strategies; and better IT operations to respond to enhanced service-level agreements. Missed (2003): Emerging non-PC markets in 2009 will account for over 40% of total HDD shipments, up from only 9.3% in 2003. The emerging non-PC markets for HDDs include, but are not limited to, automobiles, handheld portable media players, personal video recorders, game stations, video surveillance systems, cell phones and digital cameras. We were not nearly as aggressive with this prediction as were other industry pundits, but we still seriously missed the evolving mix of HDD consumption in emerging non-PC markets as opposed to the traditional notebook PC, desk-based PC, server and storage system markets. As a percentage of total mobile HDD shipments, the non-PC markets accounted for 24.8%, 33%, 30.6%, 22.1% and 19.9% in 2004, 2005, 2006, 2007 and 2008, respectively. To date in 2009, the non-PC markets consumed 13% of all mobile HDDs shipped. As a percentage of total desktop HDD shipments, the non-PC markets accounted for 10.7%, 10.6%, 11.5%, 12.5% and 13% in 2004, 2005, 2006, 2007 and 2008, respectively. To date in 2009, the non-PC markets consumed 13.7% of all desktop HDDs shipped. No enterprise HDDs are used in emerging, non-PC applications. In 2009, despite some healthy expected growth in fourth-quarter non-PC demand for HDDs, total consumption of HDDs in the non-PC markets will be less than 15% — more than 80 million units for the year, but a far cry from our 2003 prediction of 40% share of total HDD shipments. Use of HDDs in portable media players has dwindled, and cell phone use never really materialized, while the demand for HDDs in automotive and personal video recorder markets has been less than anticipated. There will be expanding new markets for HDDs, and unit shipments to these markets should effectively double to more than 150 million units per year, but we now believe the emerging non-PC applications will consume only 15% to 23% of total HDD shipments through 2013. Gartner believes the potential for greatest growth will be in personal video recorders configured in home entertainment centers. © 2009 Gartner, Inc. and/or its Affiliates. All Rights Reserved. Reproduction and distribution of this publication in any form without prior written permission is forbidden. The information contained herein has been obtained from sources believed to be reliable. Gartner disclaims all warranties as to the accuracy, completeness or adequacy of such information. Although Gartner's research may discuss legal issues related to the information technology business, Gartner does not provide legal advice or services and its research should not be construed or used as such. Gartner shall have no liability for errors, omissions or inadequacies in the information contained herein or for interpretations thereof. The opinions expressed herein are subject to change without notice.
|
