Building and Securing Kubernetes Native Products: The Pure Storage Product Security Journey

This post was co-authored by Dr. Ratinder Paul Singh Ahuja, CTO for Security and Networking, and Rajan Yadav, Director of Engineering, CNBU (Portworx), Pure Storage. Dr. Ahuja is a renowned name in the field of security and networking. 

With the demand for digital transformation increasing, enterprises rely on solution providers to not only deliver high-quality software and services but also for those solutions to be secure and compliant in the regions the enterprises operate. Every enterprise we have spoken to wants “an iPhone-like experience,” where the solution provider guarantees that running the software in their data center or in the cloud will not require deployment of any additional guardrails on top of the software they buy. 

Simply put, enterprise customers want comprehensive and mature security products that let them focus on their own use cases rather than patching solutions for security holes and vulnerabilities. 

So, how can providers meet this demand?

Shifting Architectures from Monoliths to Service Meshes

To achieve this goal, providers must follow mature security practices for building and shipping software to enterprise customers. For agility and speed, many are shifting their IT architectures, adopting containers and service meshes over traditional, monolithic infrastructures. 

Only, they’re running into a problem. Providers are hitting a wall trying to run containers in production at scale because of performance, availability, complexity, security, and cost issues. 

Those challenges are exactly what Portworx® solves for.

Portworx: From Startup to Successful Acquisition by Pure Storage

Named the #1 Kubernetes data platform by GigaOm in 2022, Portworx lets you run containers and modern databases, such as Kafka, Cassandra, and Postgres, at scale. Containers can be spun up in days rather than months, with operational costs reduced by up to 50% or more. It’s why the Global 2000 trusts Portworx to run containers and modern databases in production. But, getting there doesn’t happen overnight.

Building a startup is hard. Building a successful enterprise startup is even harder. The Portworx founders (Murli Thirumale, Vinod Jayaraman, and Gou Rao) and VP, Venkat Ramakrishnan, did an amazing job creating the #1 Kubernetes data platform. We credit the culture of innovation on the Portworx team as the main reason they earned the top spot in the market in their segment. 

But startups have to move fast to stay ahead of the competition. Sometimes, that means focusing more on big innovations than small details, like the many layers of security concerns. A product can be secure, but when a team lacks the bandwidth or the size to have a full-scale secure software development process implemented, there could be gaps.  

When Portworx was acquired by Pure Storage in October of 2020, part of that integration was an expectation of a certain level of maturity to incorporate secure development. With this priority and resources from the broader Pure Storage team, Portworx began its upgrade to an enterprise-grade secure platform. 

The DevSecOps Process at Pure Storage

Pure Storage has a formal secure software development process in place led by Dr. Ratinder Paul Singh Ahuja, CTO for Security and Networking. Upon acquisition, Portworx was to undergo this process—a detailed DevSecOps methodology, dubbed the “6-Point Plan.”  

We believe security is not an orthogonal process. Instead, security should be an integral part of the product development lifecycle in all stages: development, deployment, and operations: 

The DevSecOps methodology instituted at Pure Storage puts heavy emphasis on the “Shift Left” approach to security. This means integrating security best practices into design, code, and deployment at the earliest possible time in a product’s lifecycle.

Below, Figure 2 shows the typical development lifecycle and the security tools and processes that should be incorporated at each phase:

What Is the 6-Point Plan? A Proactive Approach to Security

To make adoption easier and more actionable for the various business units at Pure Storage, Dr. Ahuja distilled the DevSecOps process into an easy-to-understand program called the 6-Point Plan. Think of it as a framework that helps developers adopt a security-first mindset from the inception of an idea to the delivery of that idea, whether it’s a product or software.

Figure 3 depicts the pillars of the 6-Point Plan:

Most companies will face a critical security issue that affects their products and overall software development process. When products do not go through a threat modeling process, there is no defined secure software development lifecycle (SSDLC) practice in place. This lack of process leads to security weaknesses in the product and inadequate security controls in the deployed infrastructure, thus increasing the risk of exploits and data breaches.

Additionally, when security issues are treated as interruptions instead of proactively addressed, product managers have to spend time responding to customer inquiries and providing ad hoc fixes. This reactive approach is not sustainable and can lead to further problems down the line.

To resolve this issue, companies must prioritize the implementation of a robust SSDLC practice that includes threat modeling, continuous code scans, and regular vulnerability assessments. This process should be integrated into the CI/CD pipeline to ensure that security is baked into development from the start. Then, appropriate security controls in the infrastructure should be implemented to prevent unauthorized access and protect sensitive data. 

By taking these steps, companies can build a strong foundation for secure product development and ensure that their products are reliable and secure for their customers.

What follows is the path Portworx took to successfully adopt DevSecOps and embark on this journey of continuous improvement in the security posture of their products.

Phase 1: Discovery

The Portworx adoption of the Pure Storage DevSecOps process was led by Rajan Yadav, Director of Engineering. Like any good investigator, Rajan first sought to understand where Portworx stood in their development practices and where they should stand, defining the roadmap and plan to arrive at a place where they could build and ship secure software. 

• Things that were going well: The Portworx team was focused on running vulnerability scans on container images it was shipping to its customers. When vulnerabilities were discovered, they were fixed. This was a best effort but a reactive approach. Vulnerabilities could be anywhere, from infrastructure to container base images to application code.
• Things that were not going well: The reactive and ad hoc methodology meant security issues were often discovered late in the release cycle or by customers. The impact was often lost productivity, and sometimes, loss of reputation.

Phase 2: Integrating the 6-Point Plan

Next, Rajan came up with a framework to evaluate the current security posture of Portworx products and services. The goal was to transform the Software Development Lifecycle into a Secure Software Development Lifecycle by adopting the 6-Point Plan. But the challenge would be doing this without causing a major business disruption or rejection by the development teams.  

Initially, there was pushback from the engineering teams, and reasonably so. A team that was used to thinking about security as the best effort was now being asked to consider it up front. How did we overcome this hurdle?

We started with influence and alignment. First, we sat down with the product management and engineering teams, discussed the 6-point plan, and showcased the benefits of adopting the framework. Maturity levels were defined so each of the six points could be adopted in small, measurable, and achievable steps.

Once we got on the same page, the Portworx team incorporated the gaps in the process as part of releases. Thinking of security as a side project during product development was never going to work, so we put guardrails and milestones in place that included threat modeling, continuous scanning, and remediation as part of the daily software development process. 

Additionally, the team mapped critical gaps in the products and evaluated the ROI for each of the six points. This helped identify what should be done right away and what could wait. This helped the engineering teams plan their delivery timelines better and empowered them to take ownership of the entire DevSecOps process.  

Phase 3: Full Adoption and Launch

Within a span of nine months, the Portworx team at Pure not only aligned itself and adopted the SSDLC process, it also launched two new SaaS products (Portworx Data Services and Portworx Backup as a Service) that fully adhered to the 6-point plan: 

1. Threat modeling 
2. Static and dynamic scanning 
3. Configuration hardening 
4. Learning and training 
5. Infrastructure hardening 
6. Securely deploying and operating the services (on-prem and in cloud)

Note: Learning and training should not be a blocker to getting started with points 1 through 3. Product development is already in flight, customers are using the products, and there are more urgent needs to deliver secure products than learning everything about DevSecOps. 

Value Delivered

The following practices saved time and money and helped prevent loss of reputation: 

1. Formal threat modeling uncovered security weaknesses during the design phase as opposed to finding them during pentesting or getting hit with an actual exploit.
2. Scanning tools discovered vulnerabilities in open source and in custom code during the coding and build phases.
3. Regular penetration testing gave customers additional confidence.
4. Hardened infrastructure (AWS, containers, Kubernetes, development environments) reduced attack surface areas.
5. Security controls and SecOps engagement provided formal runbooks for potential intrusions.
6. Continuous training on secure coding practices improves the developer communities’ appreciation for security issues and how to avoid them early in the SSDLC process.

Key Takeaways

Overhauling an engineering culture to follow SSDLC is doable as long as there is resolve within the organization’s leadership to make it happen and a willingness to run quick experiments and learn from them.  

Some key points to remember while embarking on this journey:

1. The barrier to entry for a developer should be as low as having a secure CI pipeline and providing instant feedback on the code the developer wants to push to production. 
2. The same holds true for configuration changes one wants to make to their infrastructure, whether it is on-prem or in the cloud.
3. A lot of best practices have already been defined by the open source DevSecOps community. It’s a good place to start your security journey.
4. Last but not least, have an executive sponsor within your business unit who will champion the cause. 

In the next blog post, we’ll cover the first point of the 6-point DevSecOps plan in more detail and share how to go about implementing threat modeling for your products and features.