Jump to content

Crowdstrike Outage a/k/a Everything's Fucked


RPM

Recommended Posts

9 minutes ago, boilerhorn said:

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

So... yes then? 

I feel for the poor IT workers having to try and walk end users through this process. 

I guess the other option is just keep rebooting and hope it works. (Go network go!)

Link to comment
Share on other sites

2 minutes ago, Celery Man said:

there's a specific brand of leadership dude I've seen several times at these kinds of companies that is.... gen x guy who traded the puka shell necklace for a print button down but has changed nothing else about his style since 1998

yup. this looks is the current trend.  no way i trust someone who looks like that.

 

Link to comment
Share on other sites

Posted (edited)
23 minutes ago, boilerhorn said:

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

I had a personal system that got bricked by bitlocker when MS pushed an update to it, and it triggered the bitlocker into safety mode.  Only problem was, I had no idea what bitlocker was, and had never actually activated it, so no key had ever been generated. 

Edited by utee94
  • Hook 'Em 1
Link to comment
Share on other sites

how nice to have Ed Grimley leaping to solve the problem.

IMG_2958.png.222f52fad12e9e4992eb4ea0a841056f.png

“ I must say, we removed  the rogue pixels to the router out of the server diode’s dual facing holodragon… and let me spill news of an incredible sale on our product for the next 24 hours.”

  • Haha 5
Link to comment
Share on other sites

Posted (edited)

Someone tell this motherfucker that 2005 called and wants it's fauxhawk back. A grown ass man in 2024 is still sporting that shit, on TV no less.

Please no one tell him about frosted tips.

Edited by crash_davis
  • Hook 'Em 1
Link to comment
Share on other sites

6 minutes ago, utee94 said:

I had a personal system that got bricked by bitlocker when MS pushed an update to it, and it triggered the bitlocker into safety mode.  Only problem was, I had no idea what bitlocker was, and had never actually activated it, so no key had ever been generated. 

I had some fucked up bootloop issue  with an old personal laptop that was caused by some update. PC was out of warranty and I made a half-hearted effort to contact support as ASUS, and they washed their  hands of it. So I bought a six pack and started watching Youtube videos on my work PC. I was balls deep into the command window, just executing commands and deleting shit I had no idea about, but I was already prepared to just buy a new laptop so I didn't really care. It was a hail mary.

Surprisingly, I managed to completely unfuck my laptop after  a couple of hours and got another year out of it. I was so goddamned proud of myself, I felt like an IT god (I'm anything but that). But Bitlocker was a major pain in the ass with as many reboots as I went through.  I was cussing the motherfucker  that ever came up with it. 

Link to comment
Share on other sites

1 hour ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Well, that was the work around that my firm’s IT department suggested for everyone until they get to us.  They got to me very quickly.  You have to be in the office but it took about 3 minutes to delete the bad file and reboot.  It’s just a huge pain to go through 100’s of laptops.  And, it least in our case, it can’t be done remotely.  

Link to comment
Share on other sites

Just now, HouTex said:

Well, that was the work around that my firm’s IT department suggested for everyone until they get to us.  They got to me very quickly.  You have to be in the office but it took about 3 minutes to delete the bad file and reboot.  It’s just a huge pain to go through 100’s of laptops.  And, it least in our case, it can’t be done remotely.  

Yeah it's a pretty easy fix if you're technically proficient, but it's a manual process that requires hands-on-keyboard. And LOTS of folks aren't all that confident with even basic typing

  • Hook 'Em 1
Link to comment
Share on other sites

1 minute ago, markstanco said:

So all these websites are down but Surly is up.  It's backwards day. 

Angry-Steve-Jobs.jpg&f=1&nofb=1&ipt=4387

Just now, Pato del Muerto said:

Surly, obviously, doesn’t involve itself with security software. 

Nothing says stay away from the site than a bunch of assholes arguing about the same shit over and over and over.

  • Haha 1
Link to comment
Share on other sites

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

  • Hook 'Em 1
Link to comment
Share on other sites

Just now, Vegas64 said:

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

I gotta admit, I'm considering buying shares in Sentinel One this morning. Need to do a bit more research on who's who in that sector. 

  • Hook 'Em 1
Link to comment
Share on other sites

6 minutes ago, Vegas64 said:

Remember, this was the company who mismanaged layoffs and made national news just mere months ago. This company doesn’t seem well run…. I think competitors will start eating their lunch (if they aren’t already I know next to nothing about the industry they are in. Maybe Splunk or WIZ or someone like that are competing?)

 

I’m an idiot and old and not tech savvy so forgive me for being wrong here— it was cloudflare not crowdstrike who had the public flailing for mismanagement of a RIF.

 

Link to comment
Share on other sites

47 minutes ago, Blotto said:

Also the CEO was the CTO of McAfee in 2010 when they released an updated that made the antivirus think svchost.exe (a system file) was a virus. Bricked tens of thousands of computers.

 

 

  • Haha 1
Link to comment
Share on other sites

1 hour ago, Pimphand said:

Leave it to Austin to fuck up the globe on a Friday guess it's better they did it today than tomorrow but still way to go 512!

What starts here, changes the world.

  • Hook 'Em 3
Link to comment
Share on other sites

6 minutes ago, Pato del Muerto said:

How does a company that sounds like a hacker group get this much market share?  Cheapest option?

Exactly.  When I saw the email from our IT department I thought Crowdstrike was the code name for some evil hacker group.  

Link to comment
Share on other sites

1 minute ago, Captainant said:

Crowdstrike is (was, now lmao) a leading endpoint protection software that was used by the majority of F500 businesses. Part of that software's operating model is that they can push out new virus definition and application updates whenever they deem necessary which bypasses most IT organizations controls for patch maintenance windows. Because hey - it's a security update! Don't want to delay those right?

Well, it turns out allowing a 3rd party to push kernel level updates to your systems whenever they want isn't the best operating model! 

Crowdstrike pushed a bad package that's loaded at boot which causes a memory error known as a pagefault in the kernel space of the OS. This induces a kernel panic and complete crash. Because the bad file is loaded at boot every time, it locks up the OS into what's known as a "boot loop".

The fix for this is literally just to simply remove the offending file from the system32 directory, but doing so requires booting into safe mode (which doesn't load ANY non-critical dll's on boot) so you can actually load the OS and remove the file. Once it's deleted, you're done and fixed.

The problem is that most corporate machines have encryption software baked into the hardware called Bitlocker that requires retrieving a key from central IT to be able to boot into safe mode and delete the offending file. And it pretty much has to be done manually with hands on a keyboard, because you can't connect to it over the network because the host OS is kill

as someone who only dabbles in tech as a hobby, thanks for this. I actually understood all of that and it made sense.

  • Hook 'Em 1
Link to comment
Share on other sites

Posted (edited)

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 
 

Fortunately, the little company I work for only has 80,000 employees. 

Edited by Pato del Muerto
  • Fuck Around and Find Out 1
Link to comment
Share on other sites

Just now, Pato del Muerto said:

The easiest fix, I bet, is to send out instructions for safe mode boot and navigating to the file and deleting it. But there’s no way any company would trust employees to 1) be able to follow the instructions and 2) not make things worse now or in the future when they have access to the OS. 

Actually, the easiest "fix" right now is to just try and reboot your machine a million times. Seriously. You're hoping the network stack loads first and downloads the corrected sys file before the problem sys file is loaded. 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...