Jump to content

Crowdstrike Outage a/k/a Everything's Fucked


RPM

Recommended Posts

Posted (edited)
2 hours ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Yes but making sure you don't get a BSOD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

Edited by DanTheHorn
  • Fuck Around and Find Out 1
Link to comment
Share on other sites

1 minute ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers. ;)

Link to comment
Share on other sites

4 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

1 minute ago, boilerhorn said:

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers. ;)

You're telling me the push to prod ISN'T "user acceptance testing"?? Ah fuck I gotta send some emails...

  • Like 1
  • Haha 2
Link to comment
Share on other sites

I have guys working remotely that only have their work laptop with them and the only way they can get support is via Teams or email. Kinda tough if you are getting BSOD and don’t have admin privileges…. Gonna be a lot of big hotel bar bills…

Link to comment
Share on other sites

7 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

Link to comment
Share on other sites

7 minutes ago, PvilleStang said:

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

Link to comment
Share on other sites

3 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

  • Hook 'Em 1
Link to comment
Share on other sites

Posted (edited)

This is a rare one in that it isn't allowing machines to boot up.  I can't really remember something like this happening before.  Breaking something in Windows sure, but not causing blue screens.  Glad we don't use Crowdstrike at work.  It's gonna be a huge pain in the ass to fix this one if someone has to get their hands on every machine.

Edited by kevwun
Link to comment
Share on other sites

I just got a text on my personal phone from my (Global-500) company's IT department.  ".....and your PC may have Crashed/blue screened..."    Shit just got real.

Luckily I have two company laptops. One that is IT-managed and is turned off all but one day a month.  The other is a Company laptop with a a clean Windows install and just an authentication package and a VPN.   I did that for performance reasons since Zscaler, Crowdstrike and all the other crap my IT department installs steal about 30-40% of the CPU cycles.  I never guessed it would help with crash protection. 

 

    

  • Hook 'Em 1
Link to comment
Share on other sites

18 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

Link to comment
Share on other sites

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.
Link to comment
Share on other sites

1 minute ago, Buzzrock said:


Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.

As alluded to above, a couple of things could have occurred.  (1) there is some interaction with another recently deployed "feature" in windows - one they do not yet model (2) they do all of their QA in VMs - and there is a hardware, timing, etc. relationship.

Nevertheless, they missed it - and their name will live in infamy.

 

Link to comment
Share on other sites

Posted (edited)
6 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits, or getting outsourced to a consulting firm that's more interested in returning GREEN status than actually doing QA. Ive seen it firsthand in my customers time and time again

Edited by Captainant
  • Hook 'Em 4
  • Rage+1 1
Link to comment
Share on other sites

Just now, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

QA is also considered a lower caste in the IT/dev world.  Prod deployment QA is even lower than dev QA.

  • Hook 'Em 2
Link to comment
Share on other sites

15 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

So your saying there is a version of windows with just the right configuration that would have booted? I have read that there is a DLL file that is causing a memory failure. To me that most likely blows up regardless of windows version and configuration. Then again I am not an OS guy so I could be way off but I smell a rat.

Link to comment
Share on other sites

1 minute ago, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

 

Link to comment
Share on other sites

Just now, Shaggy3.0 said:

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

  • Hook 'Em 1
Link to comment
Share on other sites

1 minute ago, Captainant said:

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

Totally agree re: devops.  As you note, some think devops is 

git commit -b prod
git push

Link to comment
Share on other sites

Our shit started going down around 10:30. By 10:45 I had Leadership on 2 lines and they were both on different conference calls. On the bright side, all my shit was fubar so all I had to do was sit there and answer the phone. I already moved people to manually do what is normally done remote. Impressed the client, so I got that going for me.

  • Hook 'Em 1
Link to comment
Share on other sites

2 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

I've seen lots of cases over the years where after  the testing was done, a software guy decided to slip in more more change, or make some should-not-change-the-behavior type change like fixing a typo in a log message or fix the indentation in the code. Since the change shouldn't impact the final product, they manually bypass the testing process and push straight to production.   Sooner or later that type change accidently deletes a line it shouldn't or changes a 1 to a 0 and breaks shit.  It's very likely is going to turn out to be a manual override of the standard process.     

 

And as someone said above, QA is lower caste. If a senior developer really wants to override process, in most places, they can. 
  

  • Hook 'Em 1
Link to comment
Share on other sites

36 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

Bring back the USB cart of death

 

Link to comment
Share on other sites

2 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah it doesn't seem to be some esoteric fringe use case. It seems like the kind of thing that absolutely should have been  caught. 

Link to comment
Share on other sites

4 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah my virtual desktop is hosed lol. Thankfully I have local admin on my work laptop and didn't auto pull the update so I'm not completely hosed. 

Our outlook servers, VPN, and a few other core shared services are completely børked though. 

Gonna be an interesting Friday lol

Link to comment
Share on other sites

Posted (edited)
2 hours ago, HouTex said:

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

1 hour ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

16 minutes ago, Pato del Muerto said:

Does crowd strike not use their own product, and push it to their devices first as a last qc?

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Edited by crash_davis
Link to comment
Share on other sites

Posted (edited)

You would think this would seriously harm them.  Not like it's something you can deal with like causing printing problems or making another program to not work.  It's bricked every computer it's used on.

Edited by kevwun
Link to comment
Share on other sites

1 minute ago, crash_davis said:

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Just saw interview with Crowdstrike CEO saying it did not affect all Windows versions or patch levels. So it sucks to be Crowdstrike this morning.

Link to comment
Share on other sites

Had a flight to Tahiti today for a cruise leaving tomorrow at 3 pm. Been scrambling since 5 am to try and find a way to make it to Tahiti in time as everything is prepaid and non refundable at this point. I may have sort have found a solution by rebooking on several different airlines that were not affected. Unfortunately that solution costs an extra 6k and has me flying all over for the next 24 hours. 

Side note, walking in to IAH and seeing all the computer screens with the BSOD convinced me to abandon any attempt of solving this issue through United. Now waiting around at Hobby. 

  • Rage+1 3
Link to comment
Share on other sites

6 minutes ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

  • Haha 1
Link to comment
Share on other sites

5 minutes ago, immamac said:

I'm not sure how there isn't a mass customer exodus after this. The amount of revenue impact this event is going to have should put crowd strike completely out of business. 

I work for one of the Big 4. Trust me they are NOT fucking happy.

  • Rage+1 1
Link to comment
Share on other sites

VMs are the easy ones to repair, it's the end user laptops that are being a PITA right now.  Forcing safe mode when the end user is remote and doesn't understand how to use F keys is almost as fun as trying to show aggy their empty trophy case.

  • Hook 'Em 3
  • Haha 2
Link to comment
Share on other sites

Just now, Dahobbs said:

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

Well, let's start with these degenerate asshats pushing code on a Friday. Think of all the IT people who are going to have to work all weekend. Those same IT people will be sure to start looking at competitor software this weekend as they are working. Cloudstrike done fuck themselves good.

 

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...