On a quiet Saturday morning in April, the medical staff at a major US metropolitan hospital was busy preparing the facility for the potential arrival of COVID-19 patients as the pandemic spread across the country. However, when administrative staff tried to log into the facility’s IT system, they were met with an unexpected message: “This application is not available.”
Doctors and nurses saw a similar message when they attempted to access EMR records and other critical applications. The hospital’s IT department soon discovered that multiple Windows operating systems and servers had been mysteriously encrypted. Shortly after, the team spotted a small text file embedded in the file system:
“Your servers have been encrypted. If you want to decrypt them, email us and we’ll tell you how.”
At the same time, the hospital’s Pure Storage® FlashArray™ data storage system—part of a Purity Operating Environment installed four years earlier—swelled to 113% capacity. As the malware replicated itself, further systems became impacted. The array had been operating with an almost 5:1 data reduction, and as the data was encrypted and rewritten, it saw a 50x increase in write bandwidth until its physical space was exhausted, at which point it began to quiesce incoming writes.
Realizing they were facing a ransomware attack, and unwilling to negotiate with the hacker, the IT team promptly disabled all inbound and outbound network traffic to the data center, quarantining the encrypted servers for a criminal forensics investigation.
The array remained online and accessible, if not writable, for the duration, which allowed the security team to monitor and observe the attack in real-time while they developed a mitigation plan. IT personnel continued to monitor what was effectively a live crime scene and watched as the infrastructure strained under the load.
Formulating a Plan for the Hospital Ransomware Recovery
Because the hospital’s legacy IT infrastructure was configured as a single data center built on a single array with backups running on the same network, it wasn’t possible for administrators to access copies of system data. This rendered the hospital’s IT systems and applications unusable. As a result, staff defaulted to emergency backup procedures, consisting of paper and telephone-based communications and processes while the situation was being resolved.
In the meantime, the hospital’s IT department contacted Pure Storage Customer Support. Time was of the essence: The hospital couldn’t afford to run on emergency procedures for long.
The Pure support team responded immediately and pointed out that the IT team could access usable copies of its applications and data sets via Pure Storage snapshots, a no-cost feature residing on the FlashArray. Pure Storage snapshots are read-only snapshots of backup data and associated metadata catalogs created after full backup is performed. They provide an immutable copy of data that a ransomware attacker cannot compromise, alter, or affect. For the hospital, this meant that network data could be recovered after all, enabling its systems to be potentially rebuilt in days or hours versus weeks, accelerating a return to normal operations.
A joint team of hospital and Pure Storage personnel was formed quickly to devise a plan to add capacity to the existing array, and to add a second array to facilitate data replication and recovery. Pure Support shipped a new Pure Storage FlashArray that same day and dispatched an engineer to the hospital to work through the night to install the array and begin to replicate the data from the snapshots to the new array. The snapshots allowed the team to rapidly recreate and verify the integrity of core services such as Active Directory, DNS, and DHCP.
Restoring Critical IT Operations within Days
Within hours of the additional infrastructure arriving, the IT team brought the hospital’s restored data systems back online, bringing the emergency operating procedures to an end.
Even at the height of the attack, the hospital’s IT team found that the original FlashArray maintained its integrity and kept vital data protected despite being stretched to 113% of capacity—a testament to the resilience of the architecture.
Reflecting on the experience, the hospital’s IT manager said: “We couldn’t have gotten here without Pure Storage Customer Support and our Pure account team. With Pure’s instant response and coordinated teamwork, we got our core services back up and running within days and certified as ‘clean and fresh’ as they’ll ever be.”
He added that the Purity Operating Environment would soon be adopted by the hospital’s parent company as part of a wider IT consolidation effort, resited on a more modern network, and deployed across the entire organization.
Meeting Local Health Needs with Confidence
The knowledge and best practices gained through the ransomware recovery process and increased awareness of Pure Storage snapshots on FlashArray has given the hospital’s IT managers assurance that the organization—together with Pure Support—can safeguard itself against the effects of similar cyber-security breaches in the future—ensuring that patients’ needs can continue to be put first.
