Has this happened to you? You’ve modernized your storage infrastructure to boost performance, replacing disk-based solutions with all-flash arrays and upgrading to 200G+ Ethernet switches. But after testing some workloads over the network, you’re not seeing the low latency and high throughput the storage vendor advertised for its NVMe arrays.
The reason is simple: Overall performance is only as fast as the slowest connection. The NVMe protocol was designed specifically for flash media, dropping the baggage from the SCSI protocol which was designed for slow spinning medium, but that isn’t enough. The low-latency parallel data paths that NVMe uses to boost performance within the array also needs to extend outside the array. That performance advantage needs to continue across the network to the hosts that are running your performance-hungry applications.
That’s where NVMe over Fabric (NVMe-oF) comes in.
With Purity//FA 6.4.2, FlashArray supports NVMe/TCP, the default protocol for Ethernet, the ubiquitous network found everywhere from sprawling data centers to the most humble server closets. You know, the network the internet is built on. NVMe/TCP paves the path for cloud operating models such as Pure Fusion, which allows enterprise storage to be provisioned, managed, and consumed with the simple on-demand provisioning, effortless scale, and self-management of the cloud.
Why NVMe/TCP When Pure Storage Already Offers NVMe/FC and NVMe/RoCE?
Pure Storage has led the way with NVMe-oF. In 2019, Pure was the first vendor to support NVMe-oF on its arrays with NVMe/RoCE with data services. Then, in 2021, we extended the NVMe-oF support to Fibre Channel. And now with NVMe/TCP, organizations get all the benefits of a NVMe fabric without making the commitment to a dedicated storage network like Fibre Channel, or adding specialized RoCE equipment (i.e., lossless fabric) to their Ethernet network.
NVMe/TCP delivers on the same advantages as NVMe/FC and NVMe/RoCE: increased throughput and decreased latency. NVMe/TCP also benefits from using Transport Control Protocol (TCP), which offers:
- Scalability: TCP supports widely available, easy-to-expand networks for growing needs.
- Ease of use: TCP is a “simple” protocol with common and well-known standards that IT staff are already familiar with and trained to manage.
- Reliability: TCP has always been as a reliable Ethernet transport. with features such as packet acknowledgment, packet loss recovery, congestion control, etc.
- Worldwide usage: TCP/IP is widely deployed in data centers and is especially prevalent in cloud and hybrid environments.
If you’re in the design phase of network rollout and deciding between Fibre Channel, RoCEv2, and TCP, there are many considerations to take into account. What are your goals for performance? Scalability? Reliability? Cost? Regardless of which network protocol you choose, Pure Storage is the only company to offer NVMe over Fabric on all three of these popular network choices.
Both Fibre Channel (FC) and RoCE offer the best performance and latency compared to TCP with the caveat that rolling out either network will cost more. NVMe/RoCE also uses an Ethernet network, like TCP, but configuring RoCE does require extra steps such as flow control and a lossless network design. TCP networks offer a great balance of cost, scalability, and performance with the rollout of NVMe/TCP.
What Makes NVMe/TCP More Efficient than iSCSI?
Unlike iSCSI, NVMe protocol was designed for flash media instead of spinning media. It offers extreme parallelism with thousands of queues and commands in the queues, a lighter-weight protocol, and an architecture that takes advantage of multi-core CPUs. All this makes it far superior to iSCSI. For improving iSCSI performance in an all-flash storage environment, you need to multi-path your I/O paths, sockets, and ports, but this can create complications on the target side and queueing issues on the initiator side. Even after deploying multi-path iSCSI, you will not see the performance that NVMe/TCP brings in a comparable configuration.
NVMe/TCP also increases the efficiency of the CPU. Since NVMe/TCP is multi-threaded, and the CPU has multiple cores, the CPU is able to handle more transactions per minute. As an example, when testing MySQL with the FlashArray//X 90R3, we were able to see a 20% increase in transactions per minute and a latency reduction of 30%-35% in max latency per transaction compared to iSCSI.
When we started comparing iSCSI and NVMe/TCP against various workload sizes and link saturation levels, the results spoke for themselves. With testing on the FlashArray//X 90R3 for 4K workloads, which are a better representation of the network fabric capability rather than the back end, we saw a reduction of up to 35% in latency and a 20-25% increase in throughput/IOPS. Squeezing out additional performance in an already existing network shows why we think NVMe/TCP will increase in popularity rapidly.
Where Will NVMe/TCP Provide the Greatest Benefit?
Modern day applications with strict service level agreements are driving the need for the low-latency, high-performance, on-demand networks that NVMe-oF can deliver. In particular, applications that use microservices, everything-as-a-service, or another cloud or hybrid cloud model will benefit from a more efficient storage protocol that runs on Ethernet.
The applications benefiting from NVMe/TCP include critical applications, such as:
- Database workloads such as Oracle, MySQL, and Hadoop
- Virtual environments from VMware, Portworx, and other containers
- SAP HANA and ERP and CRM systems
- Data analytics such as Tableau, Power BI, Cassandra, and MapR
Moving to NVMe/TCP means existing OLTP applications will have faster response times, so users won’t have to “click and wait” for the transaction to complete before seeing the results. Your applications will also have more headroom so that the performance won’t degrade as more users use the applications concurrently. The performance boost can enable businesses to run new workloads that require higher throughput such as data analytics and machine learning.
Ideal for Hybrid Cloud Deployments
The cost and scalability of TCP/IP have made it the network of choice for cloud providers, with iSCSI as the standard protocol for block-based storage access. With NVMe/TCP’s demonstrated performance boosts and no cost of entry on existing TCP networks, there is widespread industry movement toward adopting NVMe/TCP to replace iSCSI. This makes it especially attractive for hybrid cloud deployments, where scalability, cost-savings, flexibility, reliability, and seamless migrations are critical requirements.
iSCSI is not going away, though. In fact, you can have NVMe-oF and iSCSI host connections to the same array. NVMe/TCP also plays nicely with your existing network traffic so you can run NVMe and iSCSI concurrently on the same network without conflicts.
A Performance Boost for the Full FlashArray Product Line
Unlike some of our competitors, the benefits of NVMe/TCP are available across our full FlashArray product line, not just our performance-oriented FlashArray//XL and FlashArray//X R3 systems. So if you’re using FlashArray//C R3 for storing snapshots or running less-demanding workloads, you’ll be able to use NVMe/TCP and see benefits. And you may find that FlashArray//C R3 can meet the performance needs of a broader range of workloads at a cost that’s competitive with hybrid flash-disk arrays. Our primary storage competitors are not as comprehensive: supporting fewer protocols than Pure Storage’s three, or only offering NVMe-oF on their higher-end all-flash arrays. With Pure, you don’t have to compromise.
The Power of Choice of NVMe over Fabrics
Pure Storage has been committed to NVMe-oF since we launched FlashArray//X in 2017, designing its chassis to be “NVMe-oF ready.” Now, with Purity//FA 6.4.2, our FlashArray customers can gain the full benefits of end-to-end NVMe on whatever network topology works best for them: FC, RoCE, or TCP. But what’s so exciting about NVMe/TCP is that all of our customers, from midsized businesses to enterprises, can take their applications to the next level using the network that they know and love.
NVMe/TCP is available today for your FlashArray fleet through the self-service upgrade capability in Pure1.
Self-service Upgrades
Learn more about how to perform your own FlashArray upgrades.
