Storage vendors have been in the news of late: EMC’s sale to Dell may be getting the headlines, but other incumbent vendors have been shrinking or restructuring, and former high-growth upstarts seem to be facing new headwinds. What is going on?
Readers of this blog will be less surprised by the current upheaval. Today’s market leader in performance storage, EMC Symmetrix, was designed in the early 1990s. To put that in perspective, this was the state of the art in cell phones back then:
Over the past twenty five years, mobile phones have been completely rethought of course, thanks to Moore’s Law and the advent of the World-wide Web, touchscreens, and so on. Ditto most everything in the data center, with servers and networks getting roughly 1000X faster (thanks again Dr. Moore) and seismic shifts like virtualization and network switching.
But then there’s storage. The Symmetrix/VMAX is no outlier—most enterprise storage solutions in use today were designed 20+ years ago. Storage has been stagnant because of mechanical disk. Hard drives do not follow Moore’s Law—yes they have gotten denser, but no faster. As a result, there has been little impetus to innovate. Enter flash.
Disruption #1: Flash memory. Over the past decade, flash memory has redefined the consumer technology experience. It is the storage inside your smart phone and is used heavily in modern data centers like those of Google, Apple and Facebook. Flash is not only much faster, but also more reliable, denser and power efficient. In our view, flash will supplant hard drives for all hot and warm data. Paraphrasing Jim Gray: as a result of Moore’s Law, flash is disk and disk is tape.
Of course, flash has been used for years as a cache or tier to accelerate disk storage. However, the performance disparity between flash and disk is so great that hybrids of flash and disk perform like disk as the “long” disk operations dominate. As we remarked before, the difference is like that between traveling internationally by jet or ship, and it is pretty hard to plan a business trip if you and your colleagues don’t know which one each will get. (Perhaps this is one of the reasons why next-gen hybrid solutions have thus far not done as well as hoped in the enterprise?)
The reason of course that flash was originally limited to a cache/tier was that it was expensive. When Pure Storage launched our all-flash storage array in 2011, the price of consumer-grade multi-level cell (cMLC) flash was about four times higher than of a fast hard drive (15K rpm). Today, cMLC costs less than fast disk, the same fast disk that underpins the $24B performance-optimized (a.k.a. “Tier 1”) storage and related software market. Given its advantages in nearly every dimension, it is our view that flash has rendered performance hard drives obsolete.
So why don’t the big storage vendors simply insert flash in place of mechanical disk in their product lines? After all, that is what the solid-state disk (SSD) was designed for. To see why this is hard to get right, let’s look at a recent example from a parallel market: When Data Domain embarked on their mission to replace the tape library with a disk-based backup & archiving appliance, they did not start with twenty year old software from a tape silo. Instead, Data Domain pursued a radical rethink of backup architecture in order to derive a solution that was purpose-built for disk. With hindsight, it sounds preposterous to start with the software from an aging physical tape library in order to build a modern disk-based backup appliance. And yet with the notable exception of EMC, who acquired XtremIO and DSSD, that is precisely the fool’s errand that the incumbent storage vendors are on. Optimizing 20 year old hard drive designs for solid-state flash will likely prove more complex than the failed attempt to migrate tape-centric products to disk was: disk and tape are sequential access, while flash is random access like memory. Call this, then, the end of the performance disk era and the rise of the purpose-built all-flash array.
Disruption #2: The cloud. Cloud is a more nebulous concept than all-flash (pun intended), but it is having a similarly disruptive impact on the storage industry. Amazon Web Services (AWS) has simply reset the bar for IT by making it dramatically easier for developers to deploy and scale applications. As a result, all data centers aspire to be cloud-like—to meld together commodity hardware with software and automation that delivers agility, elasticity, resiliency, security and most of all, simplicity (by shedding decades of accumulated complexity in storage and networking).
Generally the term public cloud is used to refer to Infrastructure/Platform as a Service (IaaS/PaaS) offerings like AWS, Microsoft Azure and Google Compute Engine. However, like most, we also include multi-tenant Software as a Service (SaaS) providers like Salesforce and Netsuite, as well as consumer Internet giants like Apple and Facebook (B2C SaaS). Keep in mind the success of public cloud does not mean the world is moving to a handful of data centers. Rather we see Pure’s public cloud customers, including LinkedIn, Workday, ServiceNow and Intuit, investing aggressively in their own cloud infrastructures as a core competency, convinced that they are not only saving money at the scale at which they operate, but also differentiating their products relative to the competition. (Feel free to quibble with whether SaaS is or is not public cloud, but it is a huge and fast growing market that is a sweet spot for Pure.)
Pure enjoys success in public cloud because we crafted our platform and business model for web-scale deployment and economics. (We had the benefit of principals from Google, Apple and Facebook joining the Pure team early.) In fact, for our target virtualized and database workloads, Pure Storage is comparable in costs to vanilla SSDs! How is that possible? Pure’s best-in-class deduplication and compression algorithms are averaging >5X data reduction. And Pure’s flash-optimized RAID6-style data protection scheme is about 2.5X more efficient than dual mirroring (which is standard cloud practice to protect against data loss in the event of overlapping failures). That means with server SSDs you need roughly 5 x 2.5 times the amount of flash (not counting over-provisioning) than required with the Pure platform (and of course with Pure you also get all of the data management features, performance optimization, and evergreen support to protect that investment in flash).
Even with compelling economics, it is still unlikely that Pure will be able to sell to hardcore do-it-yourselfers like Google and Amazon. We certainly agree that all business software is going SaaS as the consumer tech market (minus mobile app’s) already has, but the large majority of these providers today are building their own clouds rather than choosing to rely on AWS for core workloads. Our goal is to lead the industry in cloud-ready storage for those businesses not inclined to build their own storage and networking appliances (as Google and Amazon do).
And despite the growth of public cloud, private clouds will continue to thrive. No doubt AWS is a phenomenal growth story, but it is currently an $8B business in a $3.8 trillion worldwide IT and telecom market. Pure’s private cloud customers, like Barclays, Conoco, Nielsen, Presbyterian Hospital and Samsung, have constraints that do not allow them to move key workloads to the public cloud:
- Manufacturers need storage close to the equipment it controls;
- High frequency trading must be executed close to exchanges;
- Virtual desktop infrastructure (VDI) and other IO-intensive applications need to be close to the users it serves (thereby delivering a user experience that’s better than a laptop with a local SSD);
- Healthcare providers, financial institutions and federal agencies have additional security constraints that necessitate keeping data in data centers they control;
- Current application designs may not afford easy migration to the cloud; and so on.
So rather than move core workloads to public cloud, many of these large end users are building their own clouds for the same reasons the SaaS and consumer tech players are.
What will your storage industry look like? Expect even more major shifts in the storage industry going forward. It is high time for at least the performance storage market to join servers and networks on a Moore’s Law curve. At the same time, storage must shed the complexity, consulting overhead and unfriendly business practices that have helped make AWS so appealing. Alone, either the transition to flash or the cloud would be profoundly disruptive. Taken together, it is our view that none of the storage solutions designed for mechanical disk and the traditional data center will make the leap to the solid-state cloud.
Now the competition is on to see which storage solutions will deliver the most compelling business value in all-flash and cloud. Storage customers and partners, please do your homework, including trying out some of the new technology on your applications.
 IDC Predictions 2015: Accelerating Innovation — and Growth — on the 3rd Platform.