How did one CrowdStrike mistake stop the world?
Editor’s note: Check out our oft-updated live blog for all new developments about the Microsoft/CrowdStrike outage.
Updating your system is imperative to keeping it safe from cyber attacks and other threats. But sometimes it goes wrong — like it did late this week.
CrowdStrike, a cybersecurity company that protects companies and users from cyberattacks, made a mistake, leading to a global meltdown. Users with Windows computers saw the “blue screen of death”, flights were grounded, banks went dark, and websites shut down.
“This was an update to the software that got pushed out to its company’s clientele around the world, but particularly those that were using Windows servers, and within a certain time period,” Derrick Cogburn, a professor at American University, the Executive Director of the AU Institute on Disability and Public Policy, and the Faculty Co-Director of the Internet Governance Lab, told Mashable. “So it wasn’t everybody that uses CrowdStrike, but a pretty sizable set of the community.”
Cogburn said it affected a connected network of companies that were just trying to do the right thing and protect themselves and their users. But “when a provider like CrowdStrike has a problem with an update, it can ripple throughout the industry globally.”
“As we have raised awareness about cybersecurity, more companies and organizations have worked to protect themselves,” Cogburn said. “CrowdStrike is one of the best companies out there at protecting companies and organizations from a variety of cyberattacks.”
This was, of course, not a cyberattack — it appears to have been a mistake in an update — but these are the same kinds of issues that could arise from a cyberattack. Since CrowdStrike has positioned itself as the leading third-party company that provides safeguards against cyberthreats, many companies have adopted its services. Cogburn argues that CrowdStrike does a good job at combating those attacks — but it made one grave mistake that caused widespread mayhem. Too many companies are integrated with the same tool. When it fails, an entire global network of companies are affected.
How did one software update silence so many systems?
“The incident is a great example of the cascading failures that can occur given our relatively homogenous systems that comprise the backbone of IT infrastructure,” Gregory Falco, cybersecurity expert and assistant professor of engineering at Cornell University, said over email.
Rory Mir, the Electronic Frontier Foundation’s Associate Director of Community Organizing, told Mashable that these digital systems can’t be perfect all the time. We rely on them to safeguard our sites, but they “are going to fail at some point,” whether from deliberate attack or a simple mistake.
“The problem is that we’re really stuck in a digital monoculture, where decades of anti-competitive practices have created it so that just one system is responsible for so much of what we rely on from everything from airlines to hospitals to schools,” Mir said. “One mistake that creates a big failure, it happens, it’s an inevitability. But for it to have this sort of impact is a policy failure.”
Who does this affect most?
Every time a disaster occurs, we’re reminded that those most at risk are also those who are affected the most deeply by these kinds of systemic failures.
“Something we regularly see with any sort of system failure, things like malware attacks and data breaches, even if the nature of the failure affects everyone across the board, frankly people’s resiliency and ability to cope with these things do have a disparate impact,” Mir said. “People that have enough money to have backup systems and maybe can get another hotel so they can wait for another flight or something are more able to make it through this sort of disaster.”
Ultimately, access to technology is expensive. And knowing how technology works is, as Mir says, “privileged knowledge.”
“When you have something like this that’s so widespread, you sometimes don’t think about all of the unintended consequences,” Cogburn said. You think about airlines and TV stations, but you might not immediately think about how SNAP EBT is affected (it was shut down for hours) or food services and educational services. While some people are able to pivot easily and drive to the office instead of working from home, others don’t have that luxury.
“For people that have more limited options, if they’re relying on connected devices [and] connected services, and those are shut off they may not have the kind of flexibility to pivot into a more face-to-face environment or face-to-face space,” Cogburn said. “So I think that’s one of the ways that underserved populations are being affected.”
Smaller businesses might be hit harder than larger companies who can “weather the storm a little bit easier,” Cogburn explained, because they don’t have the same kind of resources to draw from.
Inevitably, it might lead to some people not trusting systems like CrowdStrike at all which, Cogburn argues, is “really dangerous.” Think about how often you don’t want to update your phone, but are then vulnerable to bugs and attacks — then scale that up by 100.
“You leave yourself incredibly vulnerable to the reason that the patch was developed in the first place,” Cogburn said.
How can we make sure this doesn’t happen again?
These kinds of failures are a bit of an inevitability, but their effects on society doesn’t have to be. Mir argues that the widespread nature of this issue is due to a lack of antitrust enforcement by the likes of the DOJ and state attorneys general.
“So far, antitrust laws have really been focused on lowering prices for consumers, which is great and all, but it’s also created this monoculture where it might just be one big company that offers a cheap deal, but then it becomes this huge single point of failure. And we can get this Y2K like scenario,” Mir said.
Mir is hopeful that this massive and unprecedented failure will lead to legislative change.
“This is largely a failure from the antitrust enforcers themselves — the DOJ, the FTC, the Attorneys General — but I think hopefully this disaster will be a wake up call for all of them and potentially for legislators to make sure antitrust laws are working in the consumers and for reasons beyond lowering prices,” Mir said.
Ultimately, this was an unprecedented failure. But, in some ways, we were lucky — it wasn’t a cyberattack. We might not be so lucky next time, so we need to address it now — before it’s too late.