FlashArray Now Certified with Hortonworks Data Platform v3.0.0

Today is a double whammy with two great announcements, first one is Pure Storage FlashArray with is a now certified storage with Hortonworks Data Platform v3.0.0 and the second one is we are excited to launch DirectFlash Fabric which uses NVMe-oF RoCE protocol. Our NVMe-oF protocol first implementation uses RoCE which stands for RDMA over Converged Ethernet (RDMA = Remote Direct Memory Access). FlashArray//X with DirectFlash technology delivers a great performance which exceeds SAS SSDs.

FlashArray Now Certified with Hortonworks Data Platform

Before I get into what is the importance of this certification and how we can change the big data world, here is the brief history of Hortonworks and their Hortonworks Data Platform(HDP).

Hortonworks:

Hortonworks was formed in June 2011 as an independent company, funded by $23 million venture capital from Yahoo! and Benchmark Capital. The company employs contributors to the open source software project Apache Hadoop. The Hortonworks Data Platform (HDP) product includes Apache Hadoop and is used for storing, processing, and analyzing large volumes of data. The platform is designed to deal with data from many sources and formats. The platform includes Hadoop technology such as the Hadoop Distributed File System, MapReduce, Pig, Hive, HBase, ZooKeeper, and additional components. Now Hortonworks completed its merger with Cloudera in Jan 2019.

Hortonworks Data Platform:

HDP is a very popular big data platform or at the very least it was one of the two most popular on-prem Hadoop platforms.  Hadoop is a collection of open-source applications which uses many servers to solve problems involving lots of structured and unstructured data. It is a framework based on distributed storage (HDFS) and the processing engine called MapReduce. HDFS basically breaks files into large blocks and distributes them across the data nodes in the cluster. HDFS by  It then manipulates the data in each node in parallel and reduces the data set by sending the data to the Reducer program to get the final aggregated result. HDP is enterprise-ready, open-source Apache Hadoop distribution based on a centralized architecture. HDP addresses the complete needs of data at rest, powers real-time customer applications, and delivers robust big data analytics that accelerate decision making and innovation. The latest version HDP delivers new capabilities for the enterprise to enable agile application deployment, new machine learning/deep learning workloads, real-time data warehousing, & security and governance.

Challenges with Big Data and DAS:

One of the biggest challenges of working with big data is data growth. Data grows constantly and often at a very high velocity. This is where a SAN architecture in which you separate compute from storage performs better than traditional DAS based Hadoop systems.  Traditional DAS systems are often notoriously difficult to scale and manage. In contrast, SAN is extremely easy to manage and scale. In particular, the Pure Storage FlashArray is able to scale from one a basic model to higher capacity and performance model non-disruptively. Now with FlashArray//X  DirectFlash Fabric, its performance exceeds SAS DAS SSDs performance, in terms of latency and throughput. This claim of better performance of FlashArray//X DirectFlash Fabric is based on the tests we have done comparing Apache Cassandra, MongoDB, MariaDB on FlashArray//X DirectFlash Fabric and SAS SSD DAS. For Apache Cassandra we found out that writes latency and operations per second were 30% better for Apache Cassandra based on FlashArray//X DirectFlash Fabric than Cassandra cluster on  SAS SSD DAS. This makes FlashArray//X a very good choice for the Hortonworks Data Platform.

Reference architecture:

For certification purposes, we have built a 5 node cluster as shown below with Pure Storage FlashArray//X with DirectFlash Fabric. The nodes were virtualized with VMware VSphere 6.5 and with Red Hat 7.x operating system. The HDP 3.0.0 was installed on this five node cluster and QATS was used for certification/testing. Basically, QATS (Quality Assured Testing Suite) is a product integration certification program designed to rigorously test Software, File System, Next-Gen Hardware and Containers with Hortonworks Data Platform (HDP).

The QATS suite validates the following aspects for the HDP components:

1. Complete functionality
2. High Availability
3. Security aspects
4. Kerberos
5. Ranger

For eg: For MapReduce it will test the following aspects:

Functional scenarios :

•       NN Bench Tool
•       MR Bench Tool
•       Terasort Job: Generate input with TeraGen. Sort with TeraSort.
•       JHS: for a different state of Jobs
•       Compression: For wordcount job, different types of Compression Codec for different Compression Types
•       Kerberos: Above Functional scenarios are run in a Kerberos enabled environment.

Hortonworks certified components:

The following components of HDP are certified to run on FlashArray//X.

Below is each component version and description:

Summary:

FlashArray//X, now with DirectFlash Fabric, using NVMe over RoCE,  is going to be a game changer in the Big Data world. FlashArray//X is very simple to use and deploy, which in-turn eases implementation and encourages adoption. It scales non-disruptively as your data grows and accelerates your project timelines. FlashArray//X with DirectFlash Fabric is certified with Hadoop Data Platform 3.0.0.

Pure Storage Virtual Lab

Sign up to experience the future of data storage with Pure Storage FlashArray™.

Take the Test Drive