Jump to content

Crowdstrike Outage a/k/a Everything's Fucked


RPM

Recommended Posts

Posted (edited)

https://abc7.com/post/network-issues-linked-microsoft-systems-causing-outages-businesses/15070489/

Fuck Microsoft, this seriously fucks up things at work. We were supposed to have 500 people flying out today. 

Network issues linked to Microsoft systems causing outages at businesses worldwide

 

LOS ANGELES (KABC) -- A software outage linked to Microsoft systems is impacting businesses worldwide Thursday night.

 

The issue was impacting media organizations, including ABC, as well as banks, airlines and other industries, according to reports by media outlets and technology experts.

Earlier in the day Frontier Airlines was among carriers experiencing problems, but it wasn't clear if it was the same issue that impacted a wider swath of the business world Thursday night.

 

The site Downdetector was reporting a spike in outages involving companies such as Amazon Web Services, Microsoft and even Ancestry.com.

 

American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning.

 

The alert did not specify that it was tied to the Microsoft issues.

 

The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.

 

Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.

 

At airports like Los Angeles International, social media users were reporting growing frustration among passengers stuck in the terminals.

Edited by RPM
Link to comment
Share on other sites

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

 

The last line in the that article is now:

Quote

Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.

 

  • Hook 'Em 2
Link to comment
Share on other sites

A lot of people are going to have a bad day on Friday. 

This seems to be a bad Crowdsource update that crashes windows boxes (including VMs).  The workaround being reported (as in the techissuestoday.com link above) is to boot into safe mode or WinRE and delete a file.   Eazy enough for a techie, but an airline ticket counter worker or supermarket manager isn't going to be doing that.  

I'm actually surprised, my team has windows boxes are part of our deployments to AWS, Azure, and an internal cloud, but I just checked everything is up and functioning....and my company has some type of relationship with Crowdstrike....The bad update is probably still working it's way around the world and will get to my stuff tomorrow morning. 😞

  

Link to comment
Share on other sites

  • RPM changed the title to Crowdstrike Outage a/k/a Everything's Fucked
Posted (edited)

Hi I’m currently stuck in Charlotte at the airport. Storms caused us to be delayed getting here as our layover, but that fucked us because then our connection to Austin was canceled at midnight this morning. Now waiting for some standbys and also booked a flight in SWA. Seeing which one gets us out of here first. Fuuuuuck

Edited by SubliminalHorn
.
  • Hook 'Em 2
  • Like 1
  • Rage+1 1
Link to comment
Share on other sites

1 hour ago, 0xdeadbeef said:

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

The last line in the that article is now:

 

you can still blame microsoft for the stupid shit way they handle dynamic linking, which is the root of this problem. On any sane OS, this would not bring the entire system to its knees.

  • Hook 'Em 2
Link to comment
Share on other sites

Posted (edited)
4 minutes ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

Depends on if they use crowdstrike and windows. Those aren't the only two options. There are other companies for security besides crowdstrike, and much of the business world runs on linux. But it's also not just your company, it's who they interact with as well. There are also rollback softwares which can undo this immediately. Those can probably be deployed very quickly. There might be regulatory reasons why your company might require the latest security updates.

Tl;dr: too many variables to tell.

Edited by WelfareBuysMyWeed
  • Hook 'Em 1
Link to comment
Share on other sites

Posted (edited)

Lol windows is such a fkn pile, it's wild how easily one bad push update can cripple their OS globally. 

I've seen so many outages caused on Windows fleets just from one bad update that forces you to ol yeller the whole install and restart from scratch

Edited by Captainant
  • Like 2
Link to comment
Share on other sites

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

  • Hook 'Em 1
  • Fuck Around and Find Out 1
Link to comment
Share on other sites

7 minutes ago, Buzzrock said:

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

  • Hook 'Em 1
  • Like 1
  • Rage+1 2
Link to comment
Share on other sites

2 minutes ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

  • Haha 1
Link to comment
Share on other sites

Just now, Buzzrock said:

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

Literally a bad DLL file that causes a page fault BSOD on boot

Link to comment
Share on other sites

I guess Texas DPS uses this system because we had a driver's test appointment for noon today for my oldest and that got cancelled at 7am this morning. System won't let me log back in to reschedule. 

  • Rage+1 1
Link to comment
Share on other sites

8 minutes ago, Buzzrock said:

Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC. 

It's usually deployed as part of a managed device, I know it's on my work laptop and virtual desktop as part of the whole minimum required security posture. I get the option to postpone windows and user software updates, but endpoint protection gets pushed through

 

2 minutes ago, 0xdeadbeef said:

It appears to be 3rd party kernel-mode driver, not a user level DLL.  

Username appropriate lulz

Link to comment
Share on other sites

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

Link to comment
Share on other sites

3 minutes ago, Buzzrock said:

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

There are for Linux, because it's a OS that's actually built for enterprise usage lol. Windows is basically a full reinstall to unfuck it, and from what I've been reading in /r/crowdstrike most admins are just completely locked out from remote windows systems due to the boot loop. 

 

Link to comment
Share on other sites

Agile for the win?

Happened to me last night around 1130PM CT.  Given it was not yet widespread and workarounds prepared, I followed the typical approach for this: reinstall w/ saving personal files.  Well, sily me...  I neglected to realize that a reinstallation of Windows would NOT reinstall the corporate image.  FML.

That said, even for technically savvy people, the process of fixing a corporate-image-installed will be a bit painful: request to boot into safe mode, enter the bitlocker key - that you will have to get off of your phone authenticated to your business (using mobile Edge), boot, remove a file.

BTW, writing this from my Mac.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...