Jump to content

Crowdstrike Outage a/k/a Everything's Fucked


RPM

Recommended Posts

3 minutes ago, Pato del Muerto said:

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 
 

Fortunately, the little company I work for only has 80,000 employees. 

Most employees do not have admin rights to boot into safe mode.

  • Hook 'Em 4
Link to comment
Share on other sites

Posted (edited)

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Edited by kevwun
  • Hook 'Em 7
  • Like 1
  • Haha 1
Link to comment
Share on other sites

22 minutes ago, Captainant said:

Crowdstrike is (was, now lmao) a leading endpoint protection software that was used by the majority of F500 businesses. Part of that software's operating model is that they can push out new virus definition and application updates whenever they deem necessary which bypasses most IT organizations controls for patch maintenance windows. Because hey - it's a security update! Don't want to delay those right?

Well, it turns out allowing a 3rd party to push kernel level updates to your systems whenever they want isn't the best operating model! 

Crowdstrike pushed a bad package that's loaded at boot which causes a memory error known as a pagefault in the kernel space of the OS. This induces a kernel panic and complete crash. Because the bad file is loaded at boot every time, it locks up the OS into what's known as a "boot loop".

The fix for this is literally just to simply remove the offending file from the system32 directory, but doing so requires booting into safe mode (which doesn't load ANY non-critical dll's on boot) so you can actually load the OS and remove the file. Once it's deleted, you're done and fixed.

The problem is that most corporate machines have encryption software baked into the hardware called Bitlocker that requires retrieving a key from central IT to be able to boot into safe mode and delete the offending file. And it pretty much has to be done manually with hands on a keyboard, because you can't connect to it over the network because the host OS is kill

 

edit: I should note that cloud users and extra savvy IT shops do have a quicker fix for this in simply detaching the virtual volume from a borked windows host, attaching it to a Linux host, and removing the file through simple filesystem operations. Since linux can just mount the volume directly, you can delete the file that way and move the now repaired volume back onto the windows virtual machine or instance.

 

20 minutes ago, Longhorn_Fan68 said:

as someone who only dabbles in tech as a hobby, thanks for this. I actually understood all of that and it made sense.

yep...i've been retired almost five years and it made me shudder lol

  • Hook 'Em 2
  • Haha 1
Link to comment
Share on other sites

10 minutes ago, aggie08 said:

Is this every IT manager's dream apocalyptic event? Where they get to swoop in and save the day, and all the pussy will be theirs for the taking? 

(if you're reading this, Chris, I'm just kidding; please don't lock me out of my computer)

 

esgjfa.gif

  • Haha 1
Link to comment
Share on other sites

9 minutes ago, kevwun said:

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Well, sure, when you put it that way.

Link to comment
Share on other sites

I will know my fate within the hour. My new southwest flight to Nashville arrives from Nashville and turns around. So if the Nashville-New Orleans flight is in the air around 2:00 pm coming here, I’m good to go.

Link to comment
Share on other sites

Well, let's start with these degenerate asshats pushing code on a Friday.


The famous “bowl and stroll”, named after the concept of bowling a ball and not waiting around to see it hit the pins.

Usually observed right before someone goes on vacation, but can also come in to play before weekends or major holidays.
Link to comment
Share on other sites

Posted (edited)

Their system was so antiquated and so crappy that they couldn't even track their own crews with it in real-time.  When the mess happened that shut them down, they were having to manually keep track of where their planes and crew were at.

Edited by kevwun
Link to comment
Share on other sites

2 minutes ago, F250 said:

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

Steve Austin Wwe GIF by A&E

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

Link to comment
Share on other sites

4 minutes ago, immamac said:

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

I think the message is that he's sitting at his desk now sipping from a delicious glass of single malt "I told you so."  It goes down so smooth.....

  • Hook 'Em 5
  • Haha 2
Link to comment
Share on other sites

13 minutes ago, F250 said:

lulz

 

I've been in a heated battle with our executive leadership over Crowdstrike for the last couple of months. I've pushed back so hard we were close to walking away from an agreement with them when this dude showed up for a meeting in Austin last week.

Assurances were made that despite our findings their product would be stable. My objections were about the stability of Crowdstrike.

Our CTO and CSO were perusauded by this dude. Yesterday I was told to stand down in my crusade against Crowdstrike.

Today has been a good day.

Steve Austin Wwe GIF by A&E

This is epic.

How do you parlay this? Do you wait for them to come to you tail between legs? Or do you proactively send an email or eyes emoji their way?

Either way, get your flowers on the next company All Hands

  • Hook 'Em 1
Link to comment
Share on other sites

1 hour ago, mchookem said:

yep...i've been retired almost five years and it made me shudder lol

I got to live through Nimda, Code Red, and a few others, but those were easier to fix. I can’t imagine what this is like with all of the remote stuff that rarely sees an admin in person.

Link to comment
Share on other sites

26 minutes ago, immamac said:

Did you save them the outage? Or did the outage happen and now you get to say I told you so? 

No outage except in our PoC environment and some managed cloud customers that brought their own Crowdstrike license. Contract negotiations were occurring this week. Now they have been paused and things are being reconsidered. I've been on an anti-CrowdStrike crusade for awhile at work and yesterday I was ordered to drop it. Now today I've asked for Crowdstrike alternatives.

  • Hook 'Em 4
Link to comment
Share on other sites

In x86, Opcode+ModRM "00 00" is Add [EAX], AL.  That will most certainly generate the page fault reported in the BSOD.

The beauty of x86 is that screwing up a binary is fairly trivial.  A kernel mode driver filled with nulls is an easy way to crash a machine.

 

Link to comment
Share on other sites

2 minutes ago, boilerhorn said:

In x86, Opcode+ModRM "00 00" is Add [EAX], AL.  That will most certainly generate the page fault reported in the BSOD.

The beauty of x86 is that screwing up a binary is fairly trivial.  A kernel mode driver filled with nulls is an easy way to crash a machine.

 

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

  • Hook 'Em 1
  • Haha 8
Link to comment
Share on other sites

1 minute ago, Brisketexan said:

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

Can't tell if you are mocking me or thanking me or both.  Not that it really matters :)

  • Hook 'Em 2
  • Haha 1
Link to comment
Share on other sites

7 minutes ago, Brisketexan said:

Man, that's what I've been telling people ALL DAY.  Glad someone came along to back me up.

And by all day, he means "ladies, do my farts smell like chili?"

  • Haha 1
  • Fuck Around and Find Out 1
Link to comment
Share on other sites

2 hours ago, Brisketexan said:

Hair-gelled turbodouches are VERY persuasive, man.

The CEO of Crowdstrike is exactly who I picture as a sales guy at a Dodge dealership trying to sell an $80K Hellcat to a guy with a 620 credit score and a monthly income of $3500. 120 month note at 12.5%. 

  • Hook 'Em 2
  • Like 1
  • Haha 3
Link to comment
Share on other sites

4 minutes ago, Newdoc said:

The CEO of Crowdstrike is exactly who I picture as a sales guy at a Dodge dealership trying to sell an $80K Hellcat to a guy with a 620 credit score and a monthly income of $3500. 120 month note at 32.5%. 

Ftfy

  • Haha 2
Link to comment
Share on other sites

Damn, company I work at dodged a major bullet by not using Crowdstrike. This week was the Summer party that all remote employees can come to New York if they want, I chose not to. Yesterday evening was the party that last til midnight and usually has an after party. 

Our nerds had an “Everyone be calm, we don’t use it” email at 5AM. Not sure how they managed to type that through a hangover. I’ve got no clue how everyone is getting home ha

Link to comment
Share on other sites

No, this is a nightmare.  No computers work and the only way to fix them is to touch every single one.  Then the people in charge will be bitching at you about why you ever spent so much money on Crowdstrike in the first place because it's not cheap.  Then you'll have to find a new endpoint security product and they'll bitch about how much it costs.  Then you get to roll it out.  It's a shitton of work because someone at Crowdstrike was a dumbass.

Then the new product will have some issue and take down yours or all of the internet because software sucks and is written by humans.
Link to comment
Share on other sites

Posted (edited)
7 hours ago, MissingInAction said:

Sorry. I don't have a Twitter account so the embed fucked up

Fwiw, if you link to a tweet it embeds a tweet. If you link to a profile it embeds their entire Twitter timeline. So, it worked as intended, though not by you.

Edited by elfenix
Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...