Crowdstrike outage: What can we learn from this failure?

Computer monitors around the world displayed a blue screen with a sad face Friday morning.

The company at the center of the outage says it wasn’t a security incident or a cyberattack, but many are questioning how something like this could happen?

STAY IN THE KNOW

	Watch NBC10 Boston news for free, 24/7, wherever you are.
	Get Boston local news, weather forecasts, lifestyle and entertainment stories to your inbox. Sign up for NBC Boston’s newsletters.

“This is a really big mess, to be honest,” said Steve Zuromski, the vice president of Information Technology at Bridgewater State University.

Cybersecurity company CrowdStrike said Friday morning’s failure was the result of a routine update gone wrong.

Get top local stories in Boston delivered to you every morning. Sign up for NBC Boston's News Headlines newsletter.

Their software is installed on Microsoft Windows clients around the world.

“The software is designed to prevent cyberattacks and malicious activities from happening on those devices,” said Zuromski.

But the update, he said, caused Microsoft cloud computing systems to crash bringing up that dreaded blue screen.

“It’s a fairly routine update,” said Zuromski. “We do these things all of the time but unfortunately this one caused a major, major incident that’s affecting millions of people.”

The tech outage affected IT across multiple industries, wreaking havoc on airlines as they issued ground stops while the issue was resolved. Meanwhile at the MBTA service continued, but the electronic signage used to track trains and busses went down. Follow NBC10 Boston: https://instagram.com/nbc10boston https://tiktok.com/@nbc10boston https://facebook.com/NBC10Boston https://twitter.com/NBC10Boston

He added that what made this more complicated is that the repairs can only be done with a manual update.

“What technicians are having to do, at least as of now, is manually go to each work station, boot it into what’s called safe mode, which is basically a diagnostic mode, delete the file that’s causing it to crash and then reboot,” said Zuromski.

Friday morning on NBC’s Today Show, CrowdStrike’s CEO apologized for the outage.

“We're deeply sorry for the impact that we've caused to customers, to travelers, to anyone affected by this, including our company,” said George Kurtz, CEO of CrowdStrike.

Zuromski said CrowdStrike is a very reputable company.

“They have a suite of cyber security products,” he said. “Unfortunately this one got the best of them.”

Zuromski said a lot can be learned from this massive outage.

“Customers need to be prepared in the event that there is a global update or a major update like this,” he said. “How they’re going to recover from it, so make sure you dust off those incident response plans and be ready when something happens.”