Jump to content
Required Reading for any Longhorn ×

Crowdstrike Outage a/k/a Everything's Fucked

RPM

By RPM
in Daily Texan

 Share

Recommended Posts

RPM

RPM 26360

Posted
Posted (edited)

https://abc7.com/post/network-issues-linked-microsoft-systems-causing-outages-businesses/15070489/

Fuck Microsoft, this seriously fucks up things at work. We were supposed to have 500 people flying out today. 

Network issues linked to Microsoft systems causing outages at businesses worldwide

 

LOS ANGELES (KABC) -- A software outage linked to Microsoft systems is impacting businesses worldwide Thursday night.

 

The issue was impacting media organizations, including ABC, as well as banks, airlines and other industries, according to reports by media outlets and technology experts.

Earlier in the day Frontier Airlines was among carriers experiencing problems, but it wasn't clear if it was the same issue that impacted a wider swath of the business world Thursday night.

 

The site Downdetector was reporting a spike in outages involving companies such as Amazon Web Services, Microsoft and even Ancestry.com.

 

American Airlines, United and Delta have asked the FAA for global ground stop on all flights, according to an alert from the FAA on Friday morning.

 

The alert did not specify that it was tied to the Microsoft issues.

 

The FAA is telling air traffic controllers to tell airborne pilots that airlines are currently experiencing communication issues.

 

Meanwhile, flights in the air will stay in the air, but no American, United or Delta flights will take off.

 

At airports like Los Angeles International, social media users were reporting growing frustration among passengers stuck in the terminals.

Edited by RPM
Link to comment
Share on other sites
0xdeadbeef

0xdeadbeef 1652

Posted
Posted

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

 

The last line in the that article is now:

Quote

Crowdstrike, the U.S. cybersecurity company, has admitted to being responsible for the error and are working to correct it.

 

  • Hook 'Em 2
Link to comment
Share on other sites
0xdeadbeef

0xdeadbeef 1652

Posted
Posted

A lot of people are going to have a bad day on Friday. 

This seems to be a bad Crowdsource update that crashes windows boxes (including VMs).  The workaround being reported (as in the techissuestoday.com link above) is to boot into safe mode or WinRE and delete a file.   Eazy enough for a techie, but an airline ticket counter worker or supermarket manager isn't going to be doing that.  

I'm actually surprised, my team has windows boxes are part of our deployments to AWS, Azure, and an internal cloud, but I just checked everything is up and functioning....and my company has some type of relationship with Crowdstrike....The bad update is probably still working it's way around the world and will get to my stuff tomorrow morning. 😞

  

Link to comment
Share on other sites
  • RPM changed the title to Crowdstrike Outage a/k/a Everything's Fucked
SubliminalHorn

SubliminalHorn 6811

Posted
Posted (edited)

Hi I’m currently stuck in Charlotte at the airport. Storms caused us to be delayed getting here as our layover, but that fucked us because then our connection to Austin was canceled at midnight this morning. Now waiting for some standbys and also booked a flight in SWA. Seeing which one gets us out of here first. Fuuuuuck

Edited by SubliminalHorn
.
  • Hook 'Em 2
  • Like 1
  • Rage+1 1
Link to comment
Share on other sites
WelfareBuysMyWeed

WelfareBuysMyWeed 5599

Posted
Posted
1 hour ago, 0xdeadbeef said:

The headline blames Microsoft but the article says it's impacting AWS and the GCP as well.  A Microsoft problem isn't going to impact those other clouds.

The last line in the that article is now:

 

you can still blame microsoft for the stupid shit way they handle dynamic linking, which is the root of this problem. On any sane OS, this would not bring the entire system to its knees.

  • Hook 'Em 2
Link to comment
Share on other sites
  • Popular Post
WelfareBuysMyWeed

WelfareBuysMyWeed 5599

Posted
  • Popular Post
Posted (edited)
30 minutes ago, TexArcher said:

I'm not a tech guy, but this seems like an undo amount of things being fucked up due to one cybersecurity system's "error."

Is it possible that there's something more nefarious going on?

It's not that this is nefarious, it's just how it fucked up that makes it such a pain. The root of the problem is a dll which is loaded at windows boot is causing a memory error that crashes the entire system. It's not hard to fix, just remove the dll and windows will boot like normal. Nothing is destroyed, it's just in a bootlock. Unfortunately, the way windows works, is that you have to use windows to remove the dll or stop it from loading, which you normally can't do because the dll is preventing windows from loading in the first place. The correct solution is to use a special mode called Safe Mode, which prevents any non-critical components (basically any dll) from being loaded at windows boot, specifically so you can fix this stuff. Problem is that getting to safe mode and knowing which dll to delete is above the capability of most people, and on sufficiently important systems, access to safe mode would be restricted to administrators anyways. It's a super easy fix, but it's a pain in the ass to do specifically because of how windows works, so people are going to have to lug their asses physically in front of these computers to make the change. And that takes time.

Once it's fixed, it's like nothing ever happened. Nothing is permanently broken. It's just a stupid update that pushed a small piece of code that has memory errors, normally a routine thing but in this specific circumstance a huge problem. It's not nefarious, it's just shitty administrating by lots of these companies. Ideally, you wouldn't just push updates to your critical systems the moment they roll out. You'd sandbox them, test them in a controlled environment, then deploy the update to your system, for this very reason.

This isn't really a huge, evil attack. It's more like shitty IT and code deployment in the worst spot. Don't attribute malice where incompetence will do.

Edited by WelfareBuysMyWeed
  • Hook 'Em 6
  • Like 3
  • Drool 1
  • Fuck Around and Find Out 1
Link to comment
Share on other sites
WelfareBuysMyWeed

WelfareBuysMyWeed 5599

Posted
Posted (edited)
4 minutes ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

Depends on if they use crowdstrike and windows. Those aren't the only two options. There are other companies for security besides crowdstrike, and much of the business world runs on linux. But it's also not just your company, it's who they interact with as well. There are also rollback softwares which can undo this immediately. Those can probably be deployed very quickly. There might be regulatory reasons why your company might require the latest security updates.

Tl;dr: too many variables to tell.

Edited by WelfareBuysMyWeed
  • Hook 'Em 1
Link to comment
Share on other sites
Captainant

Captainant 27828

Posted
Posted (edited)

Lol windows is such a fkn pile, it's wild how easily one bad push update can cripple their OS globally. 

I've seen so many outages caused on Windows fleets just from one bad update that forces you to ol yeller the whole install and restart from scratch

Edited by Captainant
  • Like 2
Link to comment
Share on other sites
Buzzrock

Buzzrock 14949

Posted
Posted

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

  • Hook 'Em 1
  • Fuck Around and Find Out 1
Link to comment
Share on other sites
Captainant

Captainant 27828

Posted
Posted
7 minutes ago, Buzzrock said:

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

  • Hook 'Em 1
  • Like 1
  • Rage+1 2
Link to comment
Share on other sites
Buzzrock

Buzzrock 14949

Posted
Posted
2 minutes ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

  • Haha 1
Link to comment
Share on other sites
Captainant

Captainant 27828

Posted
Posted
Just now, Buzzrock said:

Is that all it was? A regular definitions update? Jeez. 
 

We pushed out a McAfee update once that identified OS files as malware and quarantined / deleted them. That was a fun day. 

Literally a bad DLL file that causes a page fault BSOD on boot

Link to comment
Share on other sites
mdmost

mdmost 43056

Posted
Posted

I guess Texas DPS uses this system because we had a driver's test appointment for noon today for my oldest and that got cancelled at 7am this morning. System won't let me log back in to reschedule. 

  • Rage+1 1
Link to comment
Share on other sites
Captainant

Captainant 27828

Posted
Posted
8 minutes ago, Buzzrock said:

Does Crowdstrike make consumer solutions or is this contained to enterprise systems? The only Windows machine in my house is the boy’s gaming PC. 

It's usually deployed as part of a managed device, I know it's on my work laptop and virtual desktop as part of the whole minimum required security posture. I get the option to postpone windows and user software updates, but endpoint protection gets pushed through

 

2 minutes ago, 0xdeadbeef said:

It appears to be 3rd party kernel-mode driver, not a user level DLL.  

Username appropriate lulz

Link to comment
Share on other sites
Buzzrock

Buzzrock 14949

Posted
Posted

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

Link to comment
Share on other sites
  • Popular Post
Celery Man

Celery Man 21003

Posted
  • Popular Post
Posted
40 minutes ago, HouTex said:

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

are you the guy who kept opening the test phishing links at work>?

34 minutes ago, Buzzrock said:

It’s puzzling how this could have been rolled out so broadly without being tested and caught. Back when I did that sort of thing for a living (thankfully those days are way behind me) we tested pretty extensively before rolling out patches / updates.

image.png.c75ff6749e5281563923ea8f6744c0e4.png

  • Hook 'Em 1
  • Like 1
  • Haha 18
Link to comment
Share on other sites
Captainant

Captainant 27828

Posted
Posted
3 minutes ago, Buzzrock said:

Are there systems management tools that can automate the process of booting to safe mode and removing the offending file? In the olden days we just had to stand in front of the console or maybe get in via a remote access card one by one. 

There are for Linux, because it's a OS that's actually built for enterprise usage lol. Windows is basically a full reinstall to unfuck it, and from what I've been reading in /r/crowdstrike most admins are just completely locked out from remote windows systems due to the boot loop. 

 

Link to comment
Share on other sites
boilerhorn

boilerhorn 1399

Posted
Posted

Agile for the win?

Happened to me last night around 1130PM CT.  Given it was not yet widespread and workarounds prepared, I followed the typical approach for this: reinstall w/ saving personal files.  Well, sily me...  I neglected to realize that a reinstallation of Windows would NOT reinstall the corporate image.  FML.

That said, even for technically savvy people, the process of fixing a corporate-image-installed will be a bit painful: request to boot into safe mode, enter the bitlocker key - that you will have to get off of your phone authenticated to your business (using mobile Edge), boot, remove a file.

BTW, writing this from my Mac.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing


Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business and Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Subscribe!... Donate!... Advertise... COOKIE MONSTER!

×
×
  • Create New...