Crowdstrike Outage a/k/a Everything's Fucked

[crash_davis 23,040]

3 minutes ago, Pato del Muerto said:

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 
 

Fortunately, the little company I work for only has 80,000 employees. 

Most employees do not have admin rights to boot into safe mode.

[aggie08 20,344]

Is this every IT manager's dream apocalyptic event? Where they get to swoop in and save the day, and all the pussy will be theirs for the taking? 

(if you're reading this, Chris, I'm just kidding; please don't lock me out of my computer)

[kevwun 15,255]

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Edited July 19 by kevwun

[mchookem 15,535]

22 minutes ago, Captainant said:

Crowdstrike is (was, now lmao) a leading endpoint protection software that was used by the majority of F500 businesses. Part of that software's operating model is that they can push out new virus definition and application updates whenever they deem necessary which bypasses most IT organizations controls for patch maintenance windows. Because hey - it's a security update! Don't want to delay those right?

Well, it turns out allowing a 3rd party to push kernel level updates to your systems whenever they want isn't the best operating model! 

Crowdstrike pushed a bad package that's loaded at boot which causes a memory error known as a pagefault in the kernel space of the OS. This induces a kernel panic and complete crash. Because the bad file is loaded at boot every time, it locks up the OS into what's known as a "boot loop".

The fix for this is literally just to simply remove the offending file from the system32 directory, but doing so requires booting into safe mode (which doesn't load ANY non-critical dll's on boot) so you can actually load the OS and remove the file. Once it's deleted, you're done and fixed.

The problem is that most corporate machines have encryption software baked into the hardware called Bitlocker that requires retrieving a key from central IT to be able to boot into safe mode and delete the offending file. And it pretty much has to be done manually with hands on a keyboard, because you can't connect to it over the network because the host OS is kill

 

edit: I should note that cloud users and extra savvy IT shops do have a quicker fix for this in simply detaching the virtual volume from a borked windows host, attaching it to a Linux host, and removing the file through simple filesystem operations. Since linux can just mount the volume directly, you can delete the file that way and move the now repaired volume back onto the windows virtual machine or instance.

 

20 minutes ago, Longhorn_Fan68 said:

as someone who only dabbles in tech as a hobby, thanks for this. I actually understood all of that and it made sense.

yep...i've been retired almost five years and it made me shudder lol

[Bozo_Casanova 25,822]

2 hours ago, Captainant said:

Lol I just realized my daughter has the same hair style

She needs a corncob pipe and a can of spinach. 

[Gatorubet 29,538]

14 minutes ago, crash_davis said:

Most employees do not have admin rights to boot into safe mode.

[Pimphand 5,502]

46 minutes ago, Covri said:

What starts here, changes the world.

Yeah that was exactly what I thought when it happened

[mchookem 15,535]

10 minutes ago, aggie08 said:

Is this every IT manager's dream apocalyptic event? Where they get to swoop in and save the day, and all the pussy will be theirs for the taking? 

(if you're reading this, Chris, I'm just kidding; please don't lock me out of my computer)

 

[UTexasFight 9,292]

Solarwinds.

Crowdstrike.

 

been a solid few years for Austin software companies 

[aggie08 20,344]

9 minutes ago, kevwun said:

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Well, sure, when you put it that way.

[Gatorubet 29,538]

I will know my fate within the hour. My new southwest flight to Nashville arrives from Nashville and turns around. So if the Nashville-New Orleans flight is in the air around 2:00 pm coming here, I’m good to go.

[elfenix 34,075]

6 minutes ago, Gatorubet said:

I will know my fate within the hour. My new southwest flight to Nashville arrives from Nashville and turns around. So if the Nashville-New Orleans flight is in the air around 2:00 pm coming here, I’m good to go.

 

[aggie08 20,344]

 

 

[kevwun 15,255]

The update couldn't travel through their machines because they're still sharing a single dialup connection at every airport.

[ztejas 42,775]

3 minutes ago, elfenix said:

 

Um. What?

They run on fucking 3.1?

[randomhorn 1,500]

1 minute ago, ztejas said:

Um. What?

They run on fucking 3.1?

it's a joke, son

[Dahobbs 17,180]

Just now, randomhorn said:

it's a joke, son

... Kinda. 

[ztejas 42,775]

2 minutes ago, randomhorn said:

it's a joke, son

Yeah I figured there's no way that was true.

 

 

[Texas Jeff 1,913]

Well, let's start with these degenerate asshats pushing code on a Friday.

The famous “bowl and stroll”, named after the concept of bowling a ball and not waiting around to see it hit the pins.

Usually observed right before someone goes on vacation, but can also come in to play before weekends or major holidays.

[aggie08 20,344]

Linux? All right, Southwest, didn't know you was so fancy.

[SubliminalHorn 6,377]

Made it back to Austin 😅 happy to report our plane didn’t brick while in the air

[kevwun 15,255]

Their system was so antiquated and so crappy that they couldn't even track their own crews with it in real-time.  When the mess happened that shut them down, they were having to manually keep track of where their planes and crew were at.

Edited July 19 by kevwun

[UTexasFight 9,292]

I’m already getting “would you be interested in a later flight?” messages from American for my flight…tomorrow…at 3pm

 

gulp 

[elfenix 34,075]

25 minutes ago, randomhorn said:

it's a joke, son

Yep. They actually run on a VAX having upgraded from a PDP-11 back in 2016 or so.

Edited July 19 by elfenix

[F250 28,471]

3 hours ago, Blotto said:

Lulz, look at this fucking dildo. He's been getting into his kid's hair gel again. From the looks of it, he might be drinking it. 

Saw this  earlier as well..

He's got the golden touch. 

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

[immamac 41,616]

2 minutes ago, F250 said:

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

[kevwun 15,255]

Hopefully they didn't sign anything.

[Brisketexan 103,515]

4 minutes ago, immamac said:

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

I think the message is that he's sitting at his desk now sipping from a delicious glass of single malt "I told you so."  It goes down so smooth.....

[Vegas64 640]

13 minutes ago, F250 said:

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

This is epic.

How do you parlay this? Do you wait for them to come to you tail between legs? Or do you proactively send an email or eyes emoji their way?

Either way, get your flowers on the next company All Hands

[kevwun 15,255]

The best part is his bosses were swayed by that asshat CEO.  I'd bet money that fucker can't even install Crowdstrike on one workstation.

[Brisketexan 103,515]

2 minutes ago, kevwun said:

The best part is his bosses were swayed by that asshat CEO.  I'd bet money that fucker can't even install Crowdstrike on one workstation.

Hair-gelled turbodouches are VERY persuasive, man.

[atomheartbevo 88,913]

1 hour ago, mchookem said:

yep...i've been retired almost five years and it made me shudder lol

I got to live through Nimda, Code Red, and a few others, but those were easier to fix. I can’t imagine what this is like with all of the remote stuff that rarely sees an admin in person.

[atomheartbevo 88,913]

1 hour ago, ztejas said:

Um. What?

They run on fucking 3.1?

It’s probably a reference to a problem at a French airport that ran on Windows 3.1 around 9 years ago

https://www.zdnet.com/article/a-23-year-old-windows-3-1-system-failure-crashed-paris-airport/

[Pato del Muerto 52,035]

1 hour ago, Dahobbs said:

... Kinda. 

As recently as 2015 I worked at a company who used a dos-based controlled document management software.

[Viking 295]

1 hour ago, aggie08 said:

Linux? All right, Southwest, didn't know you was so fancy.

They're only on Linux because it's free.

[Shaggy3.0 1,694]

2 minutes ago, Viking said:

They're only on Linux because it's free.

actually, it's not really free.  TCOO can skyrocket if you don't know what you're doing.

 

[Rip76 4,880]

[F250 28,471]

26 minutes ago, immamac said:

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

No outage except in our PoC environment and some managed cloud customers that brought their own Crowdstrike license. Contract negotiations were occurring this week. Now they have been paused and things are being reconsidered. I've been on an anti-CrowdStrike crusade for awhile at work and yesterday I was ordered to drop it. Now today I've asked for Crowdstrike alternatives.

[blacklab 11,818]

 

[boilerhorn 1,253]

In x86, Opcode+ModRM "00 00" is Add [EAX], AL.  That will most certainly generate the page fault reported in the BSOD.

The beauty of x86 is that screwing up a binary is fairly trivial.  A kernel mode driver filled with nulls is an easy way to crash a machine.

 

[Brisketexan 103,515]

2 minutes ago, boilerhorn said:

In x86, Opcode+ModRM "00 00" is Add [EAX], AL.  That will most certainly generate the page fault reported in the BSOD.

The beauty of x86 is that screwing up a binary is fairly trivial.  A kernel mode driver filled with nulls is an easy way to crash a machine.

 

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

[boilerhorn 1,253]

1 minute ago, Brisketexan said:

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

Can't tell if you are mocking me or thanking me or both.  Not that it really matters

[Hugo Stiglitz 23,247]

 

Look, it could have been worse.

[Blotto 23,245]

7 minutes ago, Brisketexan said:

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

And by all day, he means "ladies, do my farts smell like chili?"

[NorthLoop 13,930]

2 hours ago, F250 said:

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

You will now be fired via email so they don't have to make eye contact with you. 

[Newdoc 3,421]

2 hours ago, Brisketexan said:

Hair-gelled turbodouches are VERY persuasive, man.

The CEO of Crowdstrike is exactly who I picture as a sales guy at a Dodge dealership trying to sell an $80K Hellcat to a guy with a 620 credit score and a monthly income of $3500. 120 month note at 12.5%. 

[elfenix 34,075]

4 minutes ago, Newdoc said:

The CEO of Crowdstrike is exactly who I picture as a sales guy at a Dodge dealership trying to sell an $80K Hellcat to a guy with a 620 credit score and a monthly income of $3500. 120 month note at 32.5%. 

Ftfy

[StassneyHorn 6,429]

Damn, company I work at dodged a major bullet by not using Crowdstrike. This week was the Summer party that all remote employees can come to New York if they want, I chose not to. Yesterday evening was the party that last til midnight and usually has an after party. 

Our nerds had an “Everyone be calm, we don’t use it” email at 5AM. Not sure how they managed to type that through a hangover. I’ve got no clue how everyone is getting home ha

[1978horn 867]

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Then the new product will have some issue and take down yours or all of the internet because software sucks and is written by humans.

[elfenix 34,075]

7 hours ago, MissingInAction said:

Sorry. I don't have a Twitter account so the embed fucked up

Fwiw, if you link to a tweet it embeds a tweet. If you link to a profile it embeds their entire Twitter timeline. So, it worked as intended, though not by you.

Edited July 19 by elfenix