Connect

On July 19, 2024, a faulty software update from cybersecurity company CrowdStrike disrupted computer systems globally — affecting banks, airlines, and health systems including UVA Health. While the outage caused temporary disruptions, it also demonstrated the strength and agility of UVA Health’s response teams. 

The issue began shortly before 2 a.m. when UVA Health Information Technology (HIT) technical team members alerted the HIT Administrator on Call, Brian Shifflett, about multiple incidents of the "blue screen of death" on Windows PCs. This notorious blue screen signals a fatal system error, rendering computers unusable. Initially, only 20 cases were reported, but the situation quickly spiraled into a widespread outage that impacted the entire health system. 

Around 3:45 a.m., Veronica Brill, MSN, RN, NEA-BC, Associate Chief Nursing Officer, arrived at University Medical Center and — with the help of others — established an Incident Command Center. “If you’ve never been part of setting up a command center, it’s similar to responding to a natural disaster,” Brill explained. A dedicated Command Center phone line which team members could join, hourly check-ins, and in-person unit rounding helped facilitate a coordinated response across UVA Health. 

"We knew immediately that something significant was happening," Shifflett recalled.  

By 4:30 a.m., it was clear that this was not just a UVA Health issue, but a worldwide outage. 

Downtime Procedures and Data Ensure Uninterrupted Patient Care 

As the full scale of the outage unfolded, UVA Health implemented downtime procedures for its clinical areas, prioritizing patient safety and operational continuity. The HIT team, led by Chief Technology Officer, Zeb Elliot MBA, MSIT, identified the best solution which required physically accessing more than 17,000 affected computers to remove the defective file. This involved deploying team members to patient units and programming USB drives to facilitate the fix. 

Command Center leaders and HIT teams immediately prioritized critical areas, beginning with in-patient units and the Emergency Department (ED), followed by labs and pharmacy services. Throughout the day, an overwhelming number of HIT team members, including individuals on PTO and academic IT partners, stepped up to help get PCs in these units back online. 

Some newer team members were unfamiliar with the downtime procedures, particularly using paper forms when digital systems failed. Nursing leaders quickly organized training sessions with materials ready by 8 a.m. to ensure that clinical staff, physicians, and labs could work effectively. 

Trained as a clinician scientist, Tracey Hoke, MD, MSc, FAAP, Chief of Quality, Performance Improvement and Population Health, quickly recognized the need for clear data on patient flow and bed availability.  

Dr. Hoke coordinated with various departments — including the bed center, transfer center, emergency department, and operating rooms — to assess how many patients could be managed despite the technical limitations. By tracking numbers and providing this information to Brill, the response team reduced the gap between available beds and incoming patients, ensuring continued patient care under difficult circumstances.  

Dr. Hoke described the situation as an "alpha trauma" for the tech team: “Working in a large academic medical center, we’re used to witnessing threats to life. We typically function in a fast-paced environment. In general, our technical teams don’t normally behave like that. They are thoughtful and methodical. But HIT team members jumped right into our frenzy and understood the intensity of the work in front of us. They walked fearlessly into every inch of the building to fix the problem and ensure that our clinicians could provide the best care for our patients,” she recalled. 

Focus Shifts to Ambulatory Sites 

With in-patient and ED units operational, the focus shifted to ambulatory sites. HIT team members worked with John Bennett, Chief Ambulatory Operations Officer, and Katie Fellows, Ambulatory Access Administrator, to address issues at clinics and outpatient centers beginning Saturday morning when HIT team members went onsite to assist individual departments and organized drop-in centers for team members to bring in their PCs for repair. 

“By Friday afternoon we had developed a game plan with our HIT partners to go to over 130 clinic locations to install the fix over the weekend so all our clinics would be fully operational by Monday morning. We prioritized the clinics that had patients on Saturday first and then the others. By Saturday evening we were fully operational again. Simply incredible!  The focus, teamwork, and dedication of our HIT team was exceptional,” remarked Bennett.  

Shifflett, who worked through the weekend, described the teamwork and leadership present during the crisis: “It was amazing to see how an organization can rally around a single goal. There was no finger-pointing, no hesitation, just pure determination to get us back online,” he said. 

Brill echoed the sentiment, emphasizing the collaborative spirit. "We were all aligned around the goal, and everyone’s opinion was valued. It allowed us to make thoughtful, swift decisions," she reflected. 

Both leaders agreed that the incident revealed the resilience and dedication of UVA Health's team members. For the HIT team, the outage was a test of their ability to manage a crisis of this scale, and they rose to the occasion, tackling each issue with precision and care. 

By Monday, July 22, all ambulatory sites were fully operational, and the emergency response efforts had transitioned to normal operations. 

Robin Parkin, CPHIMS, CHCIO, Chief Information and Technology Officer, praised the exceptional collaboration: “Working closely with our operational partners, all HIT teams rallied to perform around the clock support from Friday in early morning until Friday afternoon allowing UVA Health University Medical Center and UVA Community Health hospitals to be back on Epic by mid-afternoon. Also, HIT team members and many other volunteers worked through the weekend to prepare for clinics to open on Monday morning. The close coordination with operational leaders and HIT was phenomenal and is the reason we were able to re-open our services so quickly. I am very proud of the organization's response to this system incident.”   

At UVA Community Health, ‘Everyone Jumped in to Help.’ 

Meanwhile, UVA Community Health HIT team's response to the outage involved a multi-stage process aimed at restoring systems and ensuring patient safety. Upon discovering the issue, the HIT team and key administrators established Incident Command Centers at Prince William, Haymarket, and Culpeper Medical Centers. By 3:40 a.m., all systems had officially switched to downtime procedures. 

After the first update at 5:45 a.m., the team learned that fixing the issue required manually rebooting each affected computer using flash drives to delete a corrupt file. The Emergency Operations Plan was activated at Alert Level Orange, and the HIT team asked leaders to assess the number of impacted computers necessary to restore basic functions. Priority lists were then developed to guide HIT teams in resolving critical areas first. 

Throughout the day, team members received regular communications, and by 3 p.m. on July 19, UVA Community Health transitioned off downtime procedures, with essential systems restored. However, many computers still required fixes. UVA Community Health HIT leaders — Levy Riley and Salomon Pleitez at Prince William Medical Center and Haymarket Medical Center and Ron Clark at Culpepper Medical Center — continued adjusting priority lists to ensure the safety of critical functions, including medication scanning and infant security. 

Michael Bednar MHA, CHPA, CHEC-III, Emergency Preparedness Manager, led Community Health’s response efforts: “Our team came together swiftly, and everyone jumped in to help. We even had non-IT team members working to identify high-priority computers and marking them with sticky notes to expedite the process. I’m honored to work alongside this dedicated team throughout UVA Community Health,” he recalled.

By Monday morning, the HIT team had addressed and resolved nearly all issues, with over 1,000 computers repaired within five days. The Community Health HIT team’s coordination and dedication ensured a swift recovery from the unprecedented outage. 

“I was incredibly impressed with how Community Health seamlessly transitioned to downtime procedures during the CrowdStrike incident. The team didn't miss a beat, ensuring safe, uninterrupted patient care despite the challenges. It's a true testament to their resilience and commitment to excellence,” Elliott observed. 

Largest Unplanned Outage at UVA Health 

For Shifflett, Brill, and Bednar who have each been at UVA Health for more than a decade, this was the largest unplanned outage they have encountered. The experience fostered a renewed sense of pride in their colleagues and the organization’s ability to handle even the most unexpected challenges. 

As Brill reflected, “One of my favorite parts of working at UVA is the opportunity to work side by side with an incredibly engaged and collaborative team. This event reinforced this sentiment as we pulled through this event, together.”