Crowdstrike Outage a/k/a Everything's Fucked

[NorthLoop 13929]

49 minutes ago, Dahobbs said:

Had a flight to Tahiti today for a cruise leaving tomorrow at 3 pm. Been scrambling since 5 am to try and find a way to make it to Tahiti in time as everything is prepaid and non refundable at this point. I may have sort have found a solution by rebooking on several different airlines that were not affected. Unfortunately that solution costs an extra 6k and has me flying all over for the next 24 hours. 

Side note, walking in to IAH and seeing all the computer screens with the BSOD convinced me to abandon any attempt of solving this issue through United. Now waiting around at Hobby. 

[Dahobbs 17180]

9 minutes ago, boilerhorn said:

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

So... yes then? 

I feel for the poor IT workers having to try and walk end users through this process. 

I guess the other option is just keep rebooting and hope it works. (Go network go!)

[Shaggy3.0 1694]

2 minutes ago, Celery Man said:

there's a specific brand of leadership dude I've seen several times at these kinds of companies that is.... gen x guy who traded the puka shell necklace for a print button down but has changed nothing else about his style since 1998

yup. this looks is the current trend.  no way i trust someone who looks like that.

 

[kevwun 15255]

I bet that guy only speaks in acronyms and buzz words, with a lot of bros sprinkled in.

Edited July 19 by kevwun

[utee94 15708]

23 minutes ago, boilerhorn said:

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

I had a personal system that got bricked by bitlocker when MS pushed an update to it, and it triggered the bitlocker into safety mode.  Only problem was, I had no idea what bitlocker was, and had never actually activated it, so no key had ever been generated. 

Edited July 19 by utee94

[Gatorubet 29538]

how nice to have Ed Grimley leaping to solve the problem.

“ I must say, we removed  the rogue pixels to the router out of the server diode’s dual facing holodragon… and let me spill news of an incredible sale on our product for the next 24 hours.”

[crash_davis 23040]

Someone tell this motherfucker that 2005 called and wants it's fauxhawk back. A grown ass man in 2024 is still sporting that shit, on TV no less.

Please no one tell him about frosted tips.

Edited July 19 by crash_davis

[kevwun 15255]

You can't practice MMA everyday without a fauxhawk bro.

[Blotto 23245]

6 minutes ago, utee94 said:

I had a personal system that got bricked by bitlocker when MS pushed an update to it, and it triggered the bitlocker into safety mode.  Only problem was, I had no idea what bitlocker was, and had never actually activated it, so no key had ever been generated. 

I had some fucked up bootloop issue  with an old personal laptop that was caused by some update. PC was out of warranty and I made a half-hearted effort to contact support as ASUS, and they washed their  hands of it. So I bought a six pack and started watching Youtube videos on my work PC. I was balls deep into the command window, just executing commands and deleting shit I had no idea about, but I was already prepared to just buy a new laptop so I didn't really care. It was a hail mary.

Surprisingly, I managed to completely unfuck my laptop after  a couple of hours and got another year out of it. I was so goddamned proud of myself, I felt like an IT god (I'm anything but that). But Bitlocker was a major pain in the ass with as many reboots as I went through.  I was cussing the motherfucker  that ever came up with it. 

[HouTex 1838]

1 hour ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Well, that was the work around that my firm’s IT department suggested for everyone until they get to us.  They got to me very quickly.  You have to be in the office but it took about 3 minutes to delete the bad file and reboot.  It’s just a huge pain to go through 100’s of laptops.  And, it least in our case, it can’t be done remotely.  

[Captainant 26902]

Just now, HouTex said:

Well, that was the work around that my firm’s IT department suggested for everyone until they get to us.  They got to me very quickly.  You have to be in the office but it took about 3 minutes to delete the bad file and reboot.  It’s just a huge pain to go through 100’s of laptops.  And, it least in our case, it can’t be done remotely.  

Yeah it's a pretty easy fix if you're technically proficient, but it's a manual process that requires hands-on-keyboard. And LOTS of folks aren't all that confident with even basic typing

[Pato del Muerto 52035]

Got to work, logged on, up popped an email saying if you aren’t connected, don’t turn on your pc.  Which I guess is helpful to the employees that have email access by phone. 

[markstanco 12425]

So all these websites are down but Surly is up.  It's backwards day. 

[gofuckyourself 8057]

16 minutes ago, Blotto said:

 

I'm pretty sure that's Pauly Shore with a faux hawk

[Pato del Muerto 52035]

Just now, markstanco said:

So all these websites are down but Surly is up.  It's backwards day. 

Surly, obviously, doesn’t involve itself with security software. 

[TexArcher 37040]

I'm not sure that purchasing cybersecurity from a guy whose hairstyle is from 15 years ago is a good idea, especially if the firm name sounds like a massive hacking campaign.

[crash_davis 23040]

1 minute ago, markstanco said:

So all these websites are down but Surly is up.  It's backwards day. 

Just now, Pato del Muerto said:

Surly, obviously, doesn’t involve itself with security software. 

Nothing says stay away from the site than a bunch of assholes arguing about the same shit over and over and over.

[Pato del Muerto 52035]

1 minute ago, crash_davis said:

Nothing says stay away from the site than a bunch of assholes arguing about the same shit over and over and over.

That’s not true!

[Brisketexan 103515]

Just now, Pato del Muerto said:

That’s not true!

Negged.

[Captainant 26902]

25 minutes ago, Blotto said:

 

Lol I just realized my daughter has the same hair style

[Hugo Stiglitz 23247]

[crash_davis 23040]

 

[Brisketexan 103515]

1 minute ago, Captainant said:

Lol I just realized my daughter has the same hair style

[Shaggy3.0 1694]

2 minutes ago, Brisketexan said:

i bet she already runs that household.

 

[Vegas64 640]

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

[Blotto 23245]

Just now, Vegas64 said:

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

I gotta admit, I'm considering buying shares in Sentinel One this morning. Need to do a bit more research on who's who in that sector. 

[Captainant 26902]

I'll be goddamned, /r/WSB actually called it perfectly. This was posted about 8 hours before CS took half the internet offline

https://www.reddit.com/r/wallstreetbets/s/U3LbD1IRFT

 

[DanTheHorn 4852]

A coworker says scammers are contacting folks saying they are from CrowdStrike and need access to their system for repairs. That was quick.

[Vegas64 640]

6 minutes ago, Vegas64 said:

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

I’m an idiot and old and not tech savvy so forgive me for being wrong here— it was cloudflare not crowdstrike who had the public flailing for mismanagement of a RIF.

 

[kevwun 15255]

8 minutes ago, Captainant said:

I'll be goddamned, /r/WSB actually called it perfectly. This was posted about 8 hours before CS took half the internet offline

https://www.reddit.com/r/wallstreetbets/s/U3LbD1IRFT

 

That guy knew about the bad update somehow.

Edited July 19 by kevwun

[ztejas 42775]

47 minutes ago, Blotto said:

Also the CEO was the CTO of McAfee in 2010 when they released an updated that made the antivirus think svchost.exe (a system file) was a virus. Bricked tens of thousands of computers.

 

 

[Gatorubet 29538]

28 minutes ago, crash_davis said:

Nothing says stay away from the site than a bunch of assholes arguing about the same shit over and over and over.

Why are you dissing the Surly business model?

[Pimphand 5502]

Leave it to Austin to fuck up the globe on a Friday guess it's better they did it today than tomorrow but still way to go 512!

[royiv 9044]

32 minutes ago, Captainant said:

Lol I just realized my daughter has the same hair style

Holy lymphedema, Popeye!

[Gatorubet 29538]

39 minutes ago, Captainant said:

Lol I just realized my daughter has the same hair style

maybe cut off her Internet access?

[Blotto 23245]

If aggy sees those  forearms, its over. 

[Pato del Muerto 52035]

I wasn’t going to be mean, but since it has begun-

[RPM 26090]

1 hour ago, Hugo Stiglitz said:

[Dahobbs 17180]

[Pato del Muerto 52035]

How does a company that sounds like a hacker group get this much market share?  Cheapest option?

[Covri 6149]

1 hour ago, Pimphand said:

Leave it to Austin to fuck up the globe on a Friday guess it's better they did it today than tomorrow but still way to go 512!

What starts here, changes the world.

[MissingInAction 24324]

[HouTex 1838]

6 minutes ago, Pato del Muerto said:

How does a company that sounds like a hacker group get this much market share?  Cheapest option?

Exactly.  When I saw the email from our IT department I thought Crowdstrike was the code name for some evil hacker group.  

[Longhorn_Fan68 33708]

so anyone got a tl;dr summary of what happened, or at least what we know to this point?

[Pato del Muerto 52035]

Anonymous anti-malware suite:  if we can’t hack it, no one can!

[Longhorn_Fan68 33708]

 

 

[Captainant 26902]

20 minutes ago, Longhorn_Fan68 said:

so anyone got a tl;dr summary of what happened, or at least what we know to this point?

Crowdstrike is (was, now lmao) a leading endpoint protection software that was used by the majority of F500 businesses. Part of that software's operating model is that they can push out new virus definition and application updates whenever they deem necessary which bypasses most IT organizations controls for patch maintenance windows. Because hey - it's a security update! Don't want to delay those right?

Well, it turns out allowing a 3rd party to push kernel level updates to your systems whenever they want isn't the best operating model! 

Crowdstrike pushed a bad package that's loaded at boot which causes a memory error known as a pagefault in the kernel space of the OS. This induces a kernel panic and complete crash. Because the bad file is loaded at boot every time, it locks up the OS into what's known as a "boot loop".

The fix for this is literally just to simply remove the offending file from the system32 directory, but doing so requires booting into safe mode (which doesn't load ANY non-critical dll's on boot) so you can actually load the OS and remove the file. Once it's deleted, you're done and fixed.

The problem is that most corporate machines have encryption software baked into the hardware called Bitlocker that requires retrieving a key from central IT to be able to boot into safe mode and delete the offending file. And it pretty much has to be done manually with hands on a keyboard, because you can't connect to it over the network because the host OS is kill

 

edit: I should note that cloud users and extra savvy IT shops do have a quicker fix for this in simply detaching the virtual volume from a borked windows host, attaching it to a Linux host, and removing the file through simple filesystem operations. Since linux can just mount the volume directly, you can delete the file that way and move the now repaired volume back onto the windows virtual machine or instance.

Edited July 19 by Captainant

[Longhorn_Fan68 33708]

1 minute ago, Captainant said:

Crowdstrike is (was, now lmao) a leading endpoint protection software that was used by the majority of F500 businesses. Part of that software's operating model is that they can push out new virus definition and application updates whenever they deem necessary which bypasses most IT organizations controls for patch maintenance windows. Because hey - it's a security update! Don't want to delay those right?

Well, it turns out allowing a 3rd party to push kernel level updates to your systems whenever they want isn't the best operating model! 

Crowdstrike pushed a bad package that's loaded at boot which causes a memory error known as a pagefault in the kernel space of the OS. This induces a kernel panic and complete crash. Because the bad file is loaded at boot every time, it locks up the OS into what's known as a "boot loop".

The fix for this is literally just to simply remove the offending file from the system32 directory, but doing so requires booting into safe mode (which doesn't load ANY non-critical dll's on boot) so you can actually load the OS and remove the file. Once it's deleted, you're done and fixed.

The problem is that most corporate machines have encryption software baked into the hardware called Bitlocker that requires retrieving a key from central IT to be able to boot into safe mode and delete the offending file. And it pretty much has to be done manually with hands on a keyboard, because you can't connect to it over the network because the host OS is kill

as someone who only dabbles in tech as a hobby, thanks for this. I actually understood all of that and it made sense.

[Pato del Muerto 52035]

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 
 

Fortunately, the little company I work for only has 80,000 employees. 

Edited July 19 by Pato del Muerto

[Dahobbs 17180]

Just now, Pato del Muerto said:

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 

Actually, the easiest "fix" right now is to just try and reboot your machine a million times. Seriously. You're hoping the network stack loads first and downloads the corrected sys file before the problem sys file is loaded.