The lifeblood of many businesses is becoming increasingly dependent on the ability to deliver products and services via software. Organisations that can transform ideas and requirements into production grade software the fastest will outperform their peers:
Simply put, the importance of delivering software into production at velocity cannot be overstated.
The Conduit from Software Development to Production
The single most important software delivery conduit into production is the continuous integration (CI) pipeline and more recently the continuous delivery (CD) pipeline. Continuous integration is the practice of merging all code changes into a central repository once a day or more. Once all code changes are merged into a single place, an automated process builds the software, deploys it to a test environment and then subjects it to a series of tests. Provisioning a virtual machine or worse still a physical machine for infrequent activities is both wasteful and anything but agile. Surely there has to be an agile and expedient solution to this problem ?.
CI/CD Build Pipeline Acceleration with All Flash Storage
Pure Storage can help accelerate CI/CD build pipelines in a number of ways:
- The Raw IO Power of All Flash Storage
any IO bound build process will benefit instantly from storage designed from the ground up to harness the full power of flash, as opposed to flash retrofitted to storage designed for the era of spinning disks. - Filtering The Test Signal from the Noise with Streaming Analytics Pipelines
sifting through the sheer volume of log files and telemetry from the test stage of a pipeline can provide a serious challenge. The FlashBlade engineering team solved this problem using its own product, allowing engineering resources to be better deployed elsewhere, as outlined here. - Testing Against Fresh Production Data
eliminate the risk of test result inaccuracy due to out of date test data by integrating database refreshes from production directly into the build pipeline. - Scaling out Build Pipelines via Kubernetes and Docker Integration
CI/CD build pipeline scale-out is achievable using build agents or nodes implemented as containers. Pure Storage facilitates this on FlashArray for Kubernetes via FlexVolume integration and support for the Docker volume driver both on FlashArray and FlashBlade.
This blog post will focus on the Pure Storage Docker volume plugin, but before going any further into how this benefits a CI/CD build pipeline, let us first dispel a popular myth that containers are light weight virtual machines:
Stateful Containers and CI/CD Build Pipelines
Containers can be used by build pipelines in a variety of ways:
- as build agents or nodes
- as deployment targets
- a combination of the two
A deployment target can be created using what is known as the “Sidecar container pattern”. In the example above, the deployment target is a database. The challenge now becomes managing container state, this is where docker volumes come in.
A Build Workflow To Illustrate The “Container Sidecar” Pattern
A Jenkins pipeline that leverages the Docker plugin can be found here on GitHub, the README.md for the repository details how to set this up. This is the flow which the pipeline implements:
- Check out a SQL Server data tools project from GitHub.
- Build the project into a deploy-able artifact.
- Start a containerized SQL Server instance with a docker volume name composed of the project name, branch and build number.
- Deployed the artifact (a DACPAC file) to the SQL Server instance running inside the container.
- Run a series of tests.
- Keep the docker volume for a successful test stage, otherwise remove it.
Diving Into The Example Build Pipeline
Lets pick out some salient parts of the pipeline script, to be found in its entirety here on GitHub. The first point of interest is assigning a name to docker volume:
|
1 2 3 4 5 6 7 8 |
pipeline { agent any environment { PORT_NUMBER = 0 SCM_PROJECT = GetScmProjectName() CONTAINER_NAME = “SQLLinux${env.BRANCH_NAME}” VOLUME_NAME = “${SCM_PROJECT}_${env.BRANCH_NAME}_${env.BUILD_NUMBER}” } |
A docker volume is of limited use without spinning up a container to use it:
|
1 2 3 4 5 6 7 |
def StartContainer() { PORT_NUMBER = GetNextFreePort() sh “docker run -v ${VOLUME_NAME}:/var/opt/mssql -e \”ACCEPT_EULA=Y\” -e \”SA_PASSWORD=P@ssword1\” –name ${CONTAINER_NAME} -d -i -p ${PORT_NUMBER}:1433 microsoft/mssql-server-linux:2017-GA; sleep 10″ sh “/opt/mssql-tools/bin/sqlcmd -S localhost,${PORT_NUMBER} -U sa -P P@ssword1 -Q \”EXEC sp_configure ‘show advanced option’, ‘1’;RECONFIGURE\”” sh “/opt/mssql-tools/bin/sqlcmd -S localhost,${PORT_NUMBER} -U sa -P P@ssword1 -Q \”EXEC sp_configure ‘clr enabled’, 1;RECONFIGURE\”” sh “/opt/mssql-tools/bin/sqlcmd -S localhost,${PORT_NUMBER} -U sa -P P@ssword1 -Q \”EXEC sp_configure ‘clr strict security’, 0;RECONFIGURE\”” } |
This is where the term ‘Sidecar’ comes into play, so called because the container is being used in the context of an entity that runs at the side of the main process.
The final stage of the pipeline is to run some simple tests, for which there exists ‘Happy’ (the tests pass) and ‘Unhappy’ (the tests fail) paths:
|
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 |
stage(‘run tests (Happy path)’) { when { expression { return params.HAPPY_PATH } } steps { bat “sqlcmd -S ${CONTAINER_HOST_IP_ADDR},${PORT_NUMBER} -U sa -P P@ssword1 -d SsdtDevOpsDemo -Q \”EXEC tSQLt.Run \’tSQLtHappyPath\’\”” bat “sqlcmd -S ${CONTAINER_HOST_IP_ADDR},${PORT_NUMBER} -U sa -P P@ssword1 -d SsdtDevOpsDemo -y0 -Q \”SET NOCOUNT ON;EXEC tSQLt.XmlResultFormatter\” -o \”${WORKSPACE}\\${SCM_PROJECT}.xml\”” junit “${SCM_PROJECT}.xml” } } stage(‘run tests (Un-happy path)’) { when { expression { return !(params.HAPPY_PATH) } } steps { bat “sqlcmd -S ${CONTAINER_HOST_IP_ADDR},${PORT_NUMBER} -U sa -P P@ssword1 -d SsdtDevOpsDemo -Q \”EXEC tSQLt.Run \’tSQLtUnhappyPath\’\”” bat “sqlcmd -S ${CONTAINER_HOST_IP_ADDR},${PORT_NUMBER} -U sa -P P@ssword1 -d SsdtDevOpsDemo -y0 -Q \”SET NOCOUNT ON;EXEC tSQLt.XmlResultFormatter\” -o \”${WORKSPACE}\\${SCM_PROJECT}.xml\”” junit “${SCM_PROJECT}.xml” } } } post { always { print ‘post: Always’ node (‘linux-agent’) { sh “docker rm -f ${CONTAINER_NAME}” } } success { // // tSQLt tests have passed, therefore take no action which should mean // that the Docker volume is available for future use // print ‘post: Success’ } unstable { print ‘post: Unstable’ node (‘linux-agent’) { sh “docker volume rm -f ${VOLUME_NAME}” } } failure { print ‘post: Failure’ } } } |
No volumes are present on the Linux node prior to the first build:
Let’s instigate a build to see what happens:
If we check what docker volumes are present on the Linux node, we see that we now have a single volume:
Lets repeat this exact same exercise for the unhappy path:
There is only the original volume present after build two has completed:
Note that the use a parameter to force the execution of the pipeline to pass or fail is purely for illustrative purposes.
Summary
Sidecar containers provide the most agile means of providing environments to deploy code to in build pipelines. Additional use cases for the stateful sidecar pattern include:
- Tagging tracking system tickets with volume names. This example can be extended to cover this using the Jira plugin for Jenkins.
- Spinning up sidecar containers using existing docker volumes based on build pipeline logic.
- Passing existing docker volumes into the pipeline as parameters.
Spinning up container state in an expedient manner is vital for fast build pipeline execution. Storage designed from the ground up for flash is the perfect platform for underpinning this activity.
