What happened during the CrowdStrike Windows Outage and What to Do Nexts ??
Windows PCs running the CrowdStrike upgrade are crashing and displaying the Blue Screen of Death. Reports state that businesses all across the world have been unable to restart. Sky News is among the companies impacted by the outage; it has been unable to broadcast.
Concerned users have reported the problem on places like Reddit, where one person wrote: “Wow, stuck in a boot loop, and entire org taken out.”
You are not alone, therefore, if you arrived at work this morning to find, to put it mildly, carnage. This is what transpired and what needs to happen next.
As you may have surmised, the root of the global problem is a problem with CrowdStrike. Engineers at CrowdStrike report that they are addressing the problem, which has an impact on their Falcon Sensor product. As per CrowdStrike, Falcon is “the CrowdStrike platform designed to thwart intrusions through an integrated array of cloud-delivered security tools that thwart malware and other threats.”
Airports, companies, and broadcasters have been impacted by the IT outage, as stated on the Sky News website. In the United States, trains are affected, and boarding scanners at Edinburgh Airport in Scotland are stopped.
Microsoft says it is taking “mitigation actions” after service issues it said started at about 6pm Eastern Time. The company says it is investigating issues with cloud services in the U.S. and “an issue impacting several of its apps and services,” Sky News says.
While initial reports focused on a dodgy update, a user named Brody, who is director of CrowdStrike Overwatch posted on X, formerly Twitter that it is “a faulty channel file, so not quite an update
There is a workaround, he added.
- Boot Windows into Safe Mode or WRE.
- Go to C:\Windows\System32\drivers\CrowdStrike
- Locate and delete file matching “C-00000291*.sys”
- Boot normally.
How to Proceed
Although there is a workaround, it is not scalable and would require manual application system by system, therefore it is difficult to determine what should be done next. This could imply that it takes hours or more for a large organization to restart operations.
According to Adam Harrison, managing director at FTI Cybersecurity, if computers are in a reboot loop, the problem will naturally be very difficult to fix. System administrators will need some time to implement manual fixes because CrowdStrike is unable to remotely deploy an update. Every system will require manual intervention.
The majority won’t have anything that supports going back to known good states, but if you’re lucky, you might be able to do so, according to Harrison. “A lot of people are going to have a rough day at work when you scale that up to thousands of servers and/or thousands of workstations. The fix itself is quick to perform.”
CrowdStrike is likely to have a terrible day as well. How can the company assist individuals?
Harrison states, “They can only spread that fix as fast and widely as they can.” “It seems likely that the update has already been pulled, so any systems that hadn’t updated for whatever reason shouldn’t be subjected to a malicious update.”
Ian Thornton-Trump, CISO at Cyjax says CrowdStrike “will certainly do their very best to pull the update and instruct the old agents not to update till they can get it sorted.”
However, he says, “what has been done can not be undone for those blue screen machines. If the machines can be booted in safe mode they may be able to issue an out of band update or patch. That’s time consuming—if the machines are critical, they might actually consider restoring from backup or a shadow copy (a built in MSFT recovery feature). Whatever path they have, they will try and fix as quickly as possible.”
CrowdStrike might be able to put a tool together that would apply the fix at the disk level, such as bootable media, says Harrison. “This would maybe help some people out who have thousands of systems to fix. It’s still not a solution that solves the problem fully remotely or at huge scale, but it could bring recovery times down.”
This narrative is shocking. For updates, keep a look out and return to Macksofy blogs for more details.