Connect

Dear Team Members,

I want to provide you with a quick update on the ongoing efforts of our HIT team to restore computers across the health system.

Today’s Progress

Our efforts have continued with a focus on several key areas:

• Onsite Support: Teams have remained deployed at each of our four hospitals, prioritizing the allocation of resources to support patient care. Thankfully, requests for support have declined significantly since yesterday. 
• Ambulatory Clinics: All Ambulatory Clinics have been operational since Monday morning. We have continued to deploy rapid response teams to ensure these sites are fully supported. The needs of our Ambulatory Clinics have also declined significantly.
• Drop-In Locations: We maintained operations at nine drop-in locations throughout the health system today. Traffic was much slower at many of these sites compared to Monday. Please note that as of end of day (Tuesday) we have 20 PCs that were left at the University Hospital Dining Conference room for repair. This work has been completed and voicemails have been left to coordinate pickup.
• Virtual Support: We extended instructions to several thousand additional remote users on how to attempt recovery of their affected devices from home. This initiative aims to provide support to those outside the reach of our drop-in locations.
• Network Recovery: Approximately 96% of our computers are now live on the network, a significant improvement that reflects our team's dedication and effort to serve our team members and maintain health system operations.
• ServiceNow Tickets: All available HIT team members worked to resolve ServiceNow tickets and as of this evening, open tickets related to this incident are down to 135. Yesterday, around the same time we were at more than 300 tickets with approximately 100 additional tickets created throughout today.

Continued Support

Our in-person HIT Incident Command Center will shut down at the end of business today. HIT Incident Command Center leaders remain fully dedicated to full resolution and will direct resources to priority areas and ensure continuous communication with stakeholders.

Drop-in locations saw a significant reduction in traffic today. To ensure we are using our resources wisely, we are suspending all but one drop-in location: the University Hospital Dining Conference room will accept drop-ins tomorrow from 9 a.m. to 4 p.m.

Next Steps 

Tomorrow, our focus will be on closing out ServiceNow tickets for impacted PCs. This will streamline our ongoing efforts and allow us to concentrate on unresolved issues.

Our Rapid Response teams will remain strategically positioned to respond to issues that come in and be ready for ticket remediations as those queues are reduced. 

In some cases, computers were not recoverable and needed to reimaged. If this is the case for your computer, you may have lost software that was previously loaded. Please open a ticket with the Service Desk so that software can be replaced.

Acknowledgment

I want to express my deepest appreciation for the kindness and respect that has been displayed to my team as they have been out in the field working to restore normal operations across the UVA Health system.

Please continue to contact the HIT Service Desk at 434-924-5334 for issues.

Going forward, communications will be sent as needed. Thank you for your support. 

Warm regards,

Robin Parkin, CHCIO, CPHIMS  
Chief Information Officer  
UVA Health

Previously published on Monday, July 22

Dear Team Members,

The HIT team continues to operate its Incident Command Center to strategically deploy resources across the health system in support of computer recovery. Today’s efforts have been focused in four key areas:

• Onsite PCs across our four hospitals: We have kept teams onsite at each of our four hospitals to deploy them out of our Incident Command Center based on established priorities for supporting patient care. 
• Offsite Ambulatory Clinics: Rapid response teams continue to be deployed to offsite Ambulatory Clinics under the direction of our Incident Command Center. Since this event took place early Friday, we have worked alongside our Ambulatory Leaders to take direction on their highest priority sites. Coming into Monday morning, all Ambulatory sites were open for normal operations. Following the directions that HIT provided users, team members have appropriately entered ServiceNow tickets, and technicians have been sent to their sites to repair additional computers that were identified as critical. 
• Drop-In Locations: On Monday morning, HIT opened six drop-in locations across the Health System. These sites were busy throughout most of the morning; however, by noon HIT staff was able to reduce lines to more effectively serve our customers. In addition, HIT established two more drop-in locations, all of which will continue to serve users tomorrow (Tuesday). Please note that users On Grounds should not disconnect PC’s and bring them to our drop-in sites. Instead, please open a ServiceNow ticket for each PC impacted and include the following information:
  • Number of PC’s involved
  • Location of each PC
  • Indicate if the PC is patient-facing

On Tuesday, HIT will operate the following sites for users who are remote, or who have laptops that still require support:

• Stacey Hall – Monroe Conference Room from 9 a.m. – 4 p.m.
• UH – Dining Conference Room from 9 a.m. – 4 p.m.
• 2211 Hydraulic – Teton Conference Room from 9 a.m. – 4 p.m.
• 4105 North Fork – Lewis & Clark – Kitchen area from 9 a.m. – 4 p.m.
• Culpepper Medical Staff Office from 8 a.m. – 12 p.m.
• Prince William Women’s Center – Conference Room from 8 a.m. – 12 p.m.
• Haymarket – Admin Board Room from 8 a.m. – 12 p.m.
• Augusta – Conference Room #106 (across from retail pharmacy) from 9 a.m. – 4 p.m.
• Zion Crossroads – Second Floor Conference Room from 9 a.m. – 4 p.m.

• Virtual option for distant staff: HIT developed and deployed a set of instructions to 1,300 remote users who are known to be working on a single device (not shared). These instructions allowed users who are outside the travel radius of one of the drop-in locations to attempt to recover their impacted PC or laptop from home. HIT continues to identify an option to make this solution more universally available. 

From all reports, our hospitals have each had their HIT needs met to continue delivering care to our patients. In addition, all Ambulatory Clinics opened this morning, and our Access Teams have been able to support patients in-person and remotely.

Our data indicates that we had a normal number of users who connected to Epic today compared to any other Monday. We also have data that indicates that roughly 92% of computers were live on our network. This number is promising, as it indicates most of our computers are back online. 

Continued Communication and Support

The HIT Incident Command Center remains open with regular lines of communication to all stakeholders, and will be directing all available resources to the highest priority areas, as well as providing regular updates. 

Our Service Desk has increased staffing, but as expected has had a remarkably busy Monday. As of noon today (Monday) they had received more than 700 phone calls and at some points the wait time for an agent was up to one hour. We request that leaders continue to work with your teams to remind them to call the Service Desk only for high-priority needs that are impacting patient care. Please also try to consolidate as many issues into one call or ticket as possible to prevent multiple phone calls. Staff with laptops should attempt to take them to one of the drop-in locations for repair. Remote staff with at-home PCs should also take those to a drop-in location. While the Service Desk can log the ServiceNow ticket, they cannot provide real-time remediation.

Ongoing Preparedness

HIT intends to continue to operate its Incident Command Center to ensure as normal of operations as possible for all the UVA Health System. Please keep in mind we may continue to operate in this state for this full week while we ensure we have touched every computer that was impacted, and replaced those that were not recoverable. 

Next Steps and Actions for Employees

Please continue to encourage your teams to proactively report any issues that could affect patient care.

As a first line of support for remote workers who could not independently resolve their impacted computers, please continue to direct them to one of our drop-in locations.

If any site or team requires a higher level of support from HIT or needs additional resources, such as a drop-in resolution site, please send those communications through to our HIT Admin-on-Call, Brian Shifflett. 

Tomorrow’s focus will include closing out tickets that include a serial number of an impacted PC that was subsequently connected to our network. This will allow us to clean up our ServiceNow tickets and focus on those that truly remain untouched. 

Acknowledgment 

HIT Leadership would like to continue to express our sincere appreciation to all team members for their patience, support, and dedication during this challenging period. We would like to extend a special thank you to all our teams who stepped up and continue to work without a break to restore all impacted computers. Please know we remain confident in our collective ability to resolve this situation successfully. 

Contact Information

While the HIT Service Desk remains online and at the ready, please know they continue to experience extremely high call volumes, so your patience is appreciated. The contact details for the HIT Service Desk are 434-924-5334. 

We will continue to actively communicate with leaders and stakeholders throughout the day and provide system-wide updates, as appropriate, when we have new information. 

Thank you again for your patience and support.

Robin Parkin, CHCIO, CPHIMS
Chief Information Officer
UVA Health

Previously published on Sunday, July 21

Dear UVA Health Team Members,

UVA Health’s HIT team continues to respond to the worldwide Microsoft outage that has affected UVA Health System computers. Since this event began on Friday morning, our teams have been extremely busy and we want to keep everyone informed of the significant progress we've made as we prepare for a smooth return to operations on Monday morning. For complete updates, please see below.

Weekend Accomplishments

Computer Recovery: Our HIT team and many volunteers from other departments have been working around the clock to restore affected computers at sites across the Health System. As of now, the majority of our critical computers are back online at the following sites (listed below). Please note that this does not mean every computer is now functioning. Instead, there are a significant number of computers that have been restored in priority spaces related to patient access, patient care, nursing, and staff.

Communication and Support: Throughout the weekend, we have maintained operations of our HIT Command Center with open lines of communication to all stakeholders, providing regular updates, and directing more than 125 resources in the field. Our dedicated support teams have been available to assist with any issues that arose during this period. In addition, our Service Desk has increased staffing (when needed) to keep up with incoming calls and resolve customer needs.

Preparations for Monday

Verification from Ambulatory Leaders: To ensure readiness for Monday morning, we are working with Ambulatory leaders to complete user acceptance steps to verify that their respective sites are prepared to resume normal operations. We have asked for their assistance in confirming that all necessary preparations have been made and that their teams are ready to provide patient care without interruption on Monday morning.

Continued Monitoring: Our HIT team will continue to monitor progress closely throughout Sunday night and into Monday morning to swiftly address any residual issues that may arise. We are committed to ensuring a seamless transition back to normal business operations.

Next Steps

As we approach Monday morning, we encourage everyone to remain proactive in reporting any issues that may impact our ability to deliver exceptional care to our patients. Your support during this challenging time has been invaluable, and we have full confidence in our collective ability to navigate this situation successfully.

On-Site Employees

If you work in a patient-facing area, please check computers in your operational space. If you find a computer that is not functioning, and is required for critical patient operations, please call our HIT Help Desk at 434-924-5334 and we will open a ServiceNow ticket to be resolved as quickly as possible.

If you work at a location where multiple computers are down, please gather the serial numbers of all impacted computers and then submit a single ticket to our HIT Help Desk at 434-924-5334. By gathering tickets by department you will reduce the volume of calls coming into the Help Desk, and allow us to more effectively deploy our resources.

Remote Employees

We expect that many of our remote team members have laptop or desktop computers that may have been impacted. Please note our immediate process:

• Visit a drop-in location (if local to Virginia). UVA Health HIT will be hosting drop-in resolution sites at the following locations. We will communicate additional drop-in opportunities and hours as we move forward. 
  • Stacey Hall – Monroe Conference Room (9 a.m. - 4 p.m.)
  • UH Dining Conference Room (9 a.m. – 4 p.m.)
  • 2211 Hydraulic - Teton Conference Room (9 a.m. – 4 p.m.)
  • 4105 North Fork – Lewis & Clark – Kitchen area (9 a.m. – 4 p.m.)
  • Culpeper Medical Staff Office (9 a.m. – 5 p.m.)
  • Prince William Women’s Center Conference Room (Sunday 9 a.m. – 5 p.m., Monday 7 a.m. – 4 p.m.)
  • Augusta: conference RM Suite 106 – conference room across from retail pharmacy (9 a.m. – 4p.m.)
  • Zion Crossroads: 2nd floor conference area (9 a.m. – 4 p.m.)
• We are aware that there will still be some users who cannot take their computers to one of our drop-in locations and we are working actively to develop a plan to support those devices as we work to restore clinical operations across our Health System. Stay tuned for additional details.

Thank you to all of our team members for your patience and support as we work through this historic and unprecedented disruption. Your dedication and willingness have been outstanding, and you can count on HIT to continue its work until all computers are back online. Should you have any questions or require further assistance, please do not hesitate to reach out to our dedicated support team.

Thank you again,

Robin Parkin, CHCIO, CPHIMS
Chief Information Officer
UVA Health