Vice President of Digital Solutions Delivery for Specialty Hospitals at Mass General Brigham

Michael Ricci is the Vice President of Digital Solutions Delivery for Specialty Hospitals at Mass General Brigham, bringing over 30 years of experience in the healthcare industry. He is a visionary and goal-oriented executive known for his expertise in innovating, planning, developing, and implementing advanced information solutions to address business opportunities. Michael demonstrates strong leadership skills, fostering a collaborative team environment, and is skilled in crisis management, troubleshooting, problem-solving, business intelligence, analytics, negotiation, information security, and both hardware and software technologies.

Through this article, Ricci emphasizes that the CrowdStrike software update caused widespread issues, particularly for healthcare organizations, leading to system failures and disruptions in patient care.

On Friday, July 19, 2024, the world felt the effects of a malformed CrowdStrike software update. This update crashed Windows servers and desktops across their customer base, leaving many systems with the infamous blue screen of death (BSOD). For healthcare organizations that had taken all the right steps to protect their devices with CrowdStrike and other tools, such as BitLocker hard drive encryption, this created a perfect storm that crippled many organizations’ ability to use computers for patient care and operations. In our organization, over 45,000 devices were affected, impacting all areas of the organization. The incident led us down a path of innovation and collaboration across the enterprise to expedite remediation efforts and quickly return systems to normal operation.

  ​We not only resolved the issue for our organization in just a few days, but we also shared our solution with community hospitals outside of our system, exemplifying cooperation for the good of all patients

Initially, a manual process was developed to immediately start remediation of impacted devices. This was more than simply removing a “bad” file; it required technicians to bypass the local BitLocker hard drive encryption. This created a situation where a technician needed to read a key string, look up the corresponding 48-character numeric key password, and enter it into the device to access the local hard drive and begin the repair process. The technical teams quickly created a manual process where a technician would be at the device keyboard, phone in hand, speaking with another technical resource who looked up the key password. This was a time-consuming process, taking approximately 10-15 minutes per device. Given the incredible number of devices impacted, continuing this manual process would have taken weeks to restore all devices, which would have impacted patient care. We realized an automated process was necessary—one that could remediate a device in seconds, not minutes. It also needed to be simple enough to allow a larger population of resources to assist with deployment.

To approach a solution with automation, we assembled teams of innovators to tackle the problem from different perspectives. The first challenge was to boot the computer and access the local encrypted hard drive in an automated way. This required not only booting up the machine and avoiding the BSOD but also bypassing BitLocker encryption, which renders the drive unreadable without the BitLocker password. The second challenge was to ensure the deletion of the correct malformed file so that the computer could recover on the next reboot. All of this needed to happen quickly, with minimal user intervention, and be scalable across the enterprise. This last requirement was key, as we needed to leverage the larger workforce, not just technical desktop or server support teams. 

The approaches started with leveraging a bootable USB stick. We investigated whether Windows PE or Windows PXE would be most effective. The primary difference is that Windows PXE supports the network adapter and provides network access, whereas Windows PE does not. Given the variety of network cards in devices across the organization, we determined that Windows PE would be the most successful. With this approach, we realized that access to the BitLocker Key Password database was essential on the local USB stick. To accomplish this, we extracted the BitLocker Database from the Microsoft BitLocker Administration and Monitoring (MBAM) server and loaded it to the USB stick in an encrypted format. We then built an automated script that ran upon boot-up from the USB stick, reading the key ID from the hard drive, locating the key password, and passing it to the local disk to allow access to the drive. The script then navigated to the necessary directory and deleted the malformed file. This innovation transformed the 10-15-minute manual process into an automated process completed in under a minute. 

Once created, the collaboration needed to distribute it across the entire enterprise was awe-inspiring. People from all over the organization volunteered to take USB sticks and travel to devices across the system wherever needed. In addition to our workforce’s efforts, vendors and partners also helped by supplying additional USB sticks and USB cloning devices. The logistics involved in duplicating and delivering the USB sticks were herculean. We not only resolved the issue for our organization in just a few days, but we also shared our solution with community hospitals outside of our system, exemplifying cooperation for the good of all patients.

This experience underscored the importance of effective collaboration to respond quickly and efficiently to any healthcare crisis.