Crowdstrike Outage a/k/a Everything's Fucked

[DanTheHorn 4852]

2 hours ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Yes but making sure you don't get a BSOD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

Edited July 19 by DanTheHorn

[Buzzrock 14779]

Crowdstrike CEO on CNBC now 

[boilerhorn 1253]

1 minute ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers.

[Jkwellborn 6577]

Cue the smug Mac users

/smugmacuser

Came to ask about this. May have to go grab mine.

[Captainant 26902]

4 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

1 minute ago, boilerhorn said:

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers.

You're telling me the push to prod ISN'T "user acceptance testing"?? Ah fuck I gotta send some emails...

[Grimas 652]

I have guys working remotely that only have their work laptop with them and the only way they can get support is via Teams or email. Kinda tough if you are getting BSOD and don’t have admin privileges…. Gonna be a lot of big hotel bar bills…

[PvilleStang 2805]

7 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

[Celery Man 20568]

lol i couldn't check my kid into daycare. i mean obviously I did, but the app was just cycling and some other dad is at the touch screen going "what the hell"

[DanTheHorn 4852]

7 minutes ago, PvilleStang said:

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

[PvilleStang 2805]

3 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

[kevwun 15255]

This is a rare one in that it isn't allowing machines to boot up.  I can't really remember something like this happening before.  Breaking something in Windows sure, but not causing blue screens.  Glad we don't use Crowdstrike at work.  It's gonna be a huge pain in the ass to fix this one if someone has to get their hands on every machine.

Edited July 19 by kevwun

[TexArcher 37040]

24 minutes ago, Buzzrock said:

Former Crowdstrike CEO on CNBC now 

 

 

[Celery Man 20568]

lol up to 15 reboots may be necessary

[Shaggy3.0 1694]

5 minutes ago, TexArcher said:

 

 

did he almost die on the show?  lost his voice...sounded like shit.

chat-gpt can help update his resume.

 

[gofuckyourself 8057]

Is this because I opened that attachment about my service plan? The email was ghisj56ekdnbestbuy@yahoo.com

Seemed legit.

[0xdeadbeef 1614]

I just got a text on my personal phone from my (Global-500) company's IT department.  ".....and your PC may have Crashed/blue screened..."    Shit just got real.

Luckily I have two company laptops. One that is IT-managed and is turned off all but one day a month.  The other is a Company laptop with a a clean Windows install and just an authentication package and a VPN.   I did that for performance reasons since Zscaler, Crowdstrike and all the other crap my IT department installs steal about 30-40% of the CPU cycles.  I never guessed it would help with crash protection. 

 

    

[Blotto 23245]

18 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

[Buzzrock 14779]

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.

[boilerhorn 1253]

1 minute ago, Buzzrock said:

Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.

As alluded to above, a couple of things could have occurred.  (1) there is some interaction with another recently deployed "feature" in windows - one they do not yet model (2) they do all of their QA in VMs - and there is a hardware, timing, etc. relationship.

Nevertheless, they missed it - and their name will live in infamy.

 

[Armybrat 40527]

2 hours ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

My bank VP son is having a Bloody Mary poolside at the Bellagio in Las Vegas right now.

[Captainant 26902]

6 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits, or getting outsourced to a consulting firm that's more interested in returning GREEN status than actually doing QA. Ive seen it firsthand in my customers time and time again

Edited July 19 by Captainant

[boilerhorn 1253]

Just now, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

QA is also considered a lower caste in the IT/dev world.  Prod deployment QA is even lower than dev QA.

[DanTheHorn 4852]

15 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

So your saying there is a version of windows with just the right configuration that would have booted? I have read that there is a DLL file that is causing a memory failure. To me that most likely blows up regardless of windows version and configuration. Then again I am not an OS guy so I could be way off but I smell a rat.

[Shaggy3.0 1694]

1 minute ago, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

 

[jw4381 3044]

[Captainant 26902]

Just now, Shaggy3.0 said:

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

[WithoutAClue 14991]

[boilerhorn 1253]

1 minute ago, Captainant said:

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

Totally agree re: devops.  As you note, some think devops is 

git commit -b prod
git push

[RPM 26090]

Our shit started going down around 10:30. By 10:45 I had Leadership on 2 lines and they were both on different conference calls. On the bright side, all my shit was fubar so all I had to do was sit there and answer the phone. I already moved people to manually do what is normally done remote. Impressed the client, so I got that going for me.

[randomhorn 1500]

39 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

"worked fine on my box"

[0xdeadbeef 1614]

2 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

I've seen lots of cases over the years where after  the testing was done, a software guy decided to slip in more more change, or make some should-not-change-the-behavior type change like fixing a typo in a log message or fix the indentation in the code. Since the change shouldn't impact the final product, they manually bypass the testing process and push straight to production.   Sooner or later that type change accidently deletes a line it shouldn't or changes a 1 to a 0 and breaks shit.  It's very likely is going to turn out to be a manual override of the standard process.     

 

And as someone said above, QA is lower caste. If a senior developer really wants to override process, in most places, they can. 
  

[Pato del Muerto 52035]

Does crowd strike not use their own product, and push it to their devices first as a last qc?

[elfenix 34075]

36 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

Bring back the USB cart of death

 

[kevwun 15255]

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

[Firemans4Horn 17004]

My work email works. Everything else is FUBAR’ed

[Blotto 23245]

2 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah it doesn't seem to be some esoteric fringe use case. It seems like the kind of thing that absolutely should have been  caught. 

[Captainant 26902]

4 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah my virtual desktop is hosed lol. Thankfully I have local admin on my work laptop and didn't auto pull the update so I'm not completely hosed. 

Our outlook servers, VPN, and a few other core shared services are completely børked though. 

Gonna be an interesting Friday lol

[crash_davis 23040]

2 hours ago, HouTex said:

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

1 hour ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

16 minutes ago, Pato del Muerto said:

Does crowd strike not use their own product, and push it to their devices first as a last qc?

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Edited July 19 by crash_davis

[immamac 41615]

I'm not sure how there isn't a mass customer exodus after this. The amount of revenue impact this event is going to have should put crowd strike completely out of business. 

[ztejas 42775]

Yeah - Xbox app/store was down for me last night on my PC. 

Didn't realize it was connected to a larger issue.

Woof.

[kevwun 15255]

You would think this would seriously harm them.  Not like it's something you can deal with like causing printing problems or making another program to not work.  It's bricked every computer it's used on.

Edited July 19 by kevwun

[DanTheHorn 4852]

1 minute ago, crash_davis said:

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Just saw interview with Crowdstrike CEO saying it did not affect all Windows versions or patch levels. So it sucks to be Crowdstrike this morning.

[Dahobbs 17180]

Had a flight to Tahiti today for a cruise leaving tomorrow at 3 pm. Been scrambling since 5 am to try and find a way to make it to Tahiti in time as everything is prepaid and non refundable at this point. I may have sort have found a solution by rebooking on several different airlines that were not affected. Unfortunately that solution costs an extra 6k and has me flying all over for the next 24 hours. 

Side note, walking in to IAH and seeing all the computer screens with the BSOD convinced me to abandon any attempt of solving this issue through United. Now waiting around at Hobby. 

[kevwun 15255]

Maybe they shouldn't be testing on Windows 10 version 1803.

[Clintonaldo 2339]

LOL NERDS!

[Dahobbs 17180]

6 minutes ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

[MissingInAction 24324]

35 minutes ago, boilerhorn said:

QA is also considered a lower caste in the IT/dev world.  Prod deployment QA is even lower than dev QA.

Can confirm./notatester

[RPM 26090]

5 minutes ago, immamac said:

I'm not sure how there isn't a mass customer exodus after this. The amount of revenue impact this event is going to have should put crowd strike completely out of business. 

I work for one of the Big 4. Trust me they are NOT fucking happy.

[PvilleStang 2805]

VMs are the easy ones to repair, it's the end user laptops that are being a PITA right now.  Forcing safe mode when the end user is remote and doesn't understand how to use F keys is almost as fun as trying to show aggy their empty trophy case.

[crash_davis 23040]

Just now, Dahobbs said:

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

Well, let's start with these degenerate asshats pushing code on a Friday. Think of all the IT people who are going to have to work all weekend. Those same IT people will be sure to start looking at competitor software this weekend as they are working. Cloudstrike done fuck themselves good.