The Problem
As you may well know by now, many companies and organisations around the world, which utilised Microsoft Windows installations running CrowdStrike's endpoint security solution, Falcon, suffered massive IT service outages this week.
Banks, government departments, airports and media companies around the world were kicked in the knee terribly hard last 'Faulty Friday' as a broken security software p-code update crippled the world economy and it will still be getting back on it's feet a week from today, insiders have said.
The cause was an automatic update from CrowdStrike, not Microsoft, which deployed an update file full of zeros (0) instead of the necessary program code for detecting threats, which caused the Windows kernel to crash and halt when starting up.
The remedy is quite simple and you can have your systems back up and running in no time. However, carrying out these steps at scale can be time-consuming. If you do require help completing these steps, or if require any bespoke solutions catered specifically to your needs and your tech environment or stack, simply just give me a call and I will be glad to assist you.
The Solution
The solution requires physical access to the PC so you can do one of two things.
- Start the computer up in Safe Mode, which loads an extremely limited set of device drivers on start-up, which excludes the CrowdStrike Falcon driver which is causing the issue.
- Remove the main disk with your Windows system installed and put it into another machine to work on.
Either way, once you have gained access to your system data, you can navigate to the Windows installation folder and locate Falcon's update files, named C-00000291....sys, which need to be deleted in order to reboot your machine.
This would be a good time to get your system any data disks checked, update your Windows system and ensure you have a working backup of your current system(s) and business data. The Falcon updates should now be fixed and should be downloaded as soon as possible to reinstate your endpoint security sensors. You can never have too many backups, provided they are stored and organised well.
Warning
Please be on the lookout for scammers who say they can fix it remotely and charge you a fee without doing the work.
Also, if you are willing to perform the fix yourself, be aware that deleting the wrong files will create a different problem which will only become apparent after fixing the CrowdStrike issue.
I can fix this!
I am offering a service to fix this issue to any company or individual who is unable to at a cost of just £20 per PC.
I can visit your location(s) and remove the affected files and carry out a system check to make sure the changes rectify the issues and that the software has pulled the new updates down, ensuring your system's stability and continuity before I leave your site.
Just call me on Wakefield (01924) 919 819 to ask me any questions or to book a site visit.
Opinion
Apparently, CrowdStrike rushed out the update as a matter of importance. This caused the update file filled with zeros to be published to their update service for automatic download by their customers. The CEO seemed genuinely upset by the mistake and I don't think he should be subject to personal retribution, but I do think that the process they are using seems to be lacking the required scrutiny and checks, given the size of most of their customers - governments, multi-national enterprises, etc. The fallout, as we have now seen, has been widespread and has stopped companies dead. Ironically, given the recent push to outsource IT support, situations as widespread and diverse as this put immeasurable pressure on these companies which like to run their business right on the line, resources-wise. The best solution to this is having an in-house IT team who can get hands and brains on task as soon as the problems start. I wonder if these organisations will regret their decision to relinquish and outsource their IT knowledge.
Interestingly enough, it transpires that the current CEO of CloudStrike, George Kurtz, also presided over McAfee when they rolled out an anti-virus update which bricked Windows XP machines back in April of 2010 (https://www.zdnet.com/article/mcafee-admits-inadequate-quality-control-caused-pc-meltdown/) citing poor QA process compliance (https://www.zdnet.com/article/mcafee-admits-inadequate-quality-control-caused-pc-meltdown/.) I hope his shareholders are happy with his expense cuts ;)
Sources
Reuters article on the outage
https://www.reuters.com/technology/cybersecurity/crowdstrike-update-that-caused-global-outage-likely-skipped-checks-experts-say-2024-07-20/
A video from Dave Plummer, an ex-Microsoft software engineer
https://www.youtube.com/watch?v=wAzEJxOo1ts
A livestream discussion regarding the outage
https://www.youtube.com/watch?v=nMaIJXo33OU
Eric Parker's incisive analysis
https://www.youtube.com/watch?v=-iX1g3BVQOc
And one from Low Level Learning's channel
https://www.youtube.com/watch?v=pCxvyIx922A
A fantastic TechChrunch article on the outage
https://techcrunch.com/2024/07/19/faulty-crowdstrike-update-causes-major-global-it-outage-taking-out-banks-airlines-and-businesses-globally/
