SQL Server 2019 Big Data Clusters is a game-changing new feature for Microsoft data platform customers that provides:
This is the architecture of a SQL Server 2019 big data cluster:
A big data cluster consists of four different major components:
Each component has subtly different requirements, the compute pool – as its name suggests – is 100% compute-intensive. An infrastructure approach that allows compute and storage to be scaled independently provides the greatest degree of flexibility, commonly referred to as a disaggregated infrastructure. Similarly, a storage platform that is not tightly coupled to the same compute resources that serves the storage and data pools, allows for more flexible scaling for both storage and IO bandwidth capacity.
This can be broken down into two areas – the requirements of the Kubernetes cluster and the Big Data Cluster itself.
The state of the cluster is stored in etcd instances – etcd is a high-performance lightweight key-value store database:
The certificates that the cluster components use also constitute a state. Therefore etcd requires storage that is both durable and highly available.
The SQL Server 2019 Big Data Cluster Data is stored in two places:
The storage pool uses HDFS – “Hadoop Distributed File System” and Spark’s default storage format of Parquet. Under HDFS, data is stored in reliable distributed data sets which replicates data three times for the purposes of availability.
The storage pool uses an HDFS replication factor of 3 as per this JSON excerpt:
Data protection is built into all Pure Storage® platforms using advanced RAID and erasure coding techniques tailored specifically for flash storage, so the HDFS replication factor can, therefore, be set to 1. The data pool has the same storage requirements as that of a conventional SQL Server database.
When docker (and docker compatible) containers were first conceived, the focus was on stateless applications with objects in a Kubernetes cluster that are managed declaratively. Pods – which encapsulate containers are either part of a replicaset or a statefulset. Kubernetes aims to ensure that the number of pods specified for a replicaset or statefulset are always running. If for any reason a worker node is unable to service a workload, the Kubernetes control plane will reschedule the pods from that node to run on other worker nodes.
The rescheduling of pods in the event of a node failure for stateless applications is a trivial problem to solve:
This challenge becomes more nuanced with stateful applications, due to the fact that wherever a pod is rescheduled to run, its associated state needs to ‘Follow’ it. There are two approaches to solving this problem, the first of which is to replicate state between worker nodes:
In this scheme, if a pod needs to be rescheduled, it is placed on a worker node that its data has been replicated to. The second scheme uses a centralised storage platform:
All nodes in the cluster have an IO path to a single shared storage platform. When pods are rescheduled to run on a different worker node, their volumes are unmounted and then re-mounted to wherever they move to.
Every single piece of data that is written using a replication-based infrastructure, must be written in multiple places, which adds to the complexity and management overhead of such platforms. Modern centralised storage platforms rely on techniques such as erasure coding to make data highly available, making these both more space and hardware efficient than replication.
A big data cluster can use two types of storage; ephemeral storage or persistent volumes. With ephemeral storage, the state associated with any pod is lost the instant that pod is rescheduled to run on any other worker node in the cluster. Ephemeral storage is therefore not recommended for production purposes. All production-grade SQL Server 2019 Big Data Clusters should use persistent volumes. The Kubernetes persistent volume storage eco-system is based on three objects:
A volume is the touchpoint for storage consumption at the pod level and can be thought of in similar terms to a mount point. Storage must be associated with the volume, and this is where persistent volume claims come in – a request for storage from a storage class. Each storage entity available to the cluster is represented by a storage class. Different (or the same) storage classes can be specified for the storage and data pools, allowing storage platforms to be used which best suit the needs of each pool. For example, due to the highly parallel nature of Spark, it may be preferable to use a storage class for the storage pool associated with a platform that delivers high IO bandwidth and can service numerous IO requests concurrently.
Using a layered cake as an analogy, a volume is the topmost layer, followed by a persistent volume claim in the middle. The bottom-most layer is the actual persistent volume(s), which map directly to physical storage volumes or LUNs on the underlying storage platform.
Storage is provisioned in one of two ways:
The most basic requirements of a big data cluster are a storage platform that:
Originally all Kubernetes storage plugins were “In-tree,” meaning that all storage vendors had to integrate their drivers directly into the Kubernetes source code. This type of tight coupling is incredibly inflexible, therefore, as a direct result the Kubernetes community came up with the Flex Volume architecture. Flex Volume drivers are out-of-tree; however, a major drawback is that files must be copied onto the root file system of each node in the cluster. The state-of-the-art for Kubernetes storage driver integration is now the Container Storage Interface, often abbreviated to CSI. Not only are CSI storage plugins out-of-tree, but they are containerized and are deployed using standard Kubernetes primitives.
All storage platforms that support persistent volumes and have an associated storage class can be used for a SQL Server 2019 big data cluster, irrespective of whether the plugin is in-tree, Flex Volume or CSI compliant. However, the only Kubernetes storage plugin interface specification that has any future is the CSI standard. All storage platforms with a roadmap that aligns with the Kubernetes community, must support the Container Storage Interface standard. Pure Storage already has customers using CSI 1.0 plugins in production, we will very shortly be releasing a CSI 1.1 compliant plugin and Pure Storage is firmly committed to following future CSI developments.
Pure Storage provides a Flex Volume and CSI compliant plugin in the form of Pure Service Orchestrator™ (PSO) which provides three advanced capabilities:
In keeping with the Pure mantra of ease-of-use and simplicity, Pure Service Orchestrator is installed via Helm. The ability of Pure Service Orchestrator to elastically scale across numerous storage arrays is seamless and simply requires:
SQL Server 2019 Big Data Clusters can be deployed to vanilla Kubernetes on-premises with support for the OpenShift Container Platform set to follow. The good news is that whatever Kubernetes based platform SQL Server 2019 Big Data Clusters may support in the future, the likelihood is that Pure Service Orchestrator already supports it:
Organizations looking to deploy SQL Server 2019 Big Data Clusters are likely to be driven by the need to extract as much value as possible from their data, not from the standpoint of adding complexity and additional management overheads to their infrastructure. The good news is that we at Pure Storage make the challenge of SQL Server 2019 Big Data Cluster storage persistence an incredibly simple problem to solve. In doing so, data and analytics professionals have more time to return as much data-driven value back to their organizations as possible.
Learn best practices for managing data for Microsoft SQL Server, whether on-premises or using Cloud Block Store during the Power Up Copy Data Management for SQL Server with New Integrations Webinar, and dive deep into why and how you would want to deploy SQL Server DBs on Containers in the Streamline SQL Server Development with Containers Webinar.