Crowdstrike Outage a/k/a Everything's Fucked

[DanTheHorn 4,938]

2 hours ago, Captainant said:

Nothing breaks itself quite so eagerly as windows does. Because antivirus/endpoint protection (crowd strike) is security it gets frequent definition updates that are pushed out quickly. Since windows is a brittle cunt of an OS, one small thing sends it spiraling and more often than not requires a full OS refresh to unfuck

Yes but making sure you don't get a BSOD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

Edited July 19 by DanTheHorn

[Buzzrock 14,935]

Crowdstrike CEO on CNBC now 

[boilerhorn 1,393]

1 minute ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers.

[Jkwellborn 6,714]

Cue the smug Mac users

/smugmacuser

Came to ask about this. May have to go grab mine.

[Captainant 27,690]

4 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

1 minute ago, boilerhorn said:

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers.

You're telling me the push to prod ISN'T "user acceptance testing"?? Ah fuck I gotta send some emails...

[Grimas 670]

I have guys working remotely that only have their work laptop with them and the only way they can get support is via Teams or email. Kinda tough if you are getting BSOD and don’t have admin privileges…. Gonna be a lot of big hotel bar bills…

[PvilleStang 2,917]

7 minutes ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

[Celery Man 20,981]

lol i couldn't check my kid into daycare. i mean obviously I did, but the app was just cycling and some other dad is at the touch screen going "what the hell"

[DanTheHorn 4,938]

7 minutes ago, PvilleStang said:

That's cute, you think cybersecurity platforms wait for admins to approve patches.  

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

[PvilleStang 2,917]

3 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

[kevwun 15,811]

This is a rare one in that it isn't allowing machines to boot up.  I can't really remember something like this happening before.  Breaking something in Windows sure, but not causing blue screens.  Glad we don't use Crowdstrike at work.  It's gonna be a huge pain in the ass to fix this one if someone has to get their hands on every machine.

Edited July 19 by kevwun

[TexArcher 37,989]

24 minutes ago, Buzzrock said:

Former Crowdstrike CEO on CNBC now 

 

 

[Celery Man 20,981]

lol up to 15 reboots may be necessary

[Shaggy3.0 1,695]

5 minutes ago, TexArcher said:

 

 

did he almost die on the show?  lost his voice...sounded like shit.

chat-gpt can help update his resume.

 

[gofuckyourself 8,349]

Is this because I opened that attachment about my service plan? The email was ghisj56ekdnbestbuy@yahoo.com

Seemed legit.

[0xdeadbeef 1,651]

I just got a text on my personal phone from my (Global-500) company's IT department.  ".....and your PC may have Crashed/blue screened..."    Shit just got real.

Luckily I have two company laptops. One that is IT-managed and is turned off all but one day a month.  The other is a Company laptop with a a clean Windows install and just an authentication package and a VPN.   I did that for performance reasons since Zscaler, Crowdstrike and all the other crap my IT department installs steal about 30-40% of the CPU cycles.  I never guessed it would help with crash protection. 

 

    

[Blotto 23,560]

18 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

[Buzzrock 14,935]

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.

[boilerhorn 1,393]

1 minute ago, Buzzrock said:

Cramer asked the Crowdstrike CEO that question just now. He basically said they missed it.

As alluded to above, a couple of things could have occurred.  (1) there is some interaction with another recently deployed "feature" in windows - one they do not yet model (2) they do all of their QA in VMs - and there is a hardware, timing, etc. relationship.

Nevertheless, they missed it - and their name will live in infamy.

 

[Armybrat 40,857]

2 hours ago, TexArcher said:

Okay, so I work for a major financial firm.  Do I get today off?

My bank VP son is having a Bloody Mary poolside at the Bellagio in Las Vegas right now.

[Captainant 27,690]

6 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits, or getting outsourced to a consulting firm that's more interested in returning GREEN status than actually doing QA. Ive seen it firsthand in my customers time and time again

Edited July 19 by Captainant

[boilerhorn 1,393]

Just now, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

QA is also considered a lower caste in the IT/dev world.  Prod deployment QA is even lower than dev QA.

[DanTheHorn 4,938]

15 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

So your saying there is a version of windows with just the right configuration that would have booted? I have read that there is a DLL file that is causing a memory failure. To me that most likely blows up regardless of windows version and configuration. Then again I am not an OS guy so I could be way off but I smell a rat.

[Shaggy3.0 1,695]

1 minute ago, Captainant said:

QA is typically seen as a cost center and is often a first group targeted for downsizing by idiot empty suits

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

 

[jw4381 3,091]

[Captainant 27,690]

Just now, Shaggy3.0 said:

true.  i know several clients with 0 QA teams.  downsized and asked each domain to test their own work.

lovely

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

[WithoutAClue 15,397]

[boilerhorn 1,393]

1 minute ago, Captainant said:

Don't get me wrong, I think the devops model is better than the tradition dev/QA model, but it is rarely done "right" and not just as a buzzword thing. If QA is a defined part of your acceptance criteria then you're good, but PM's and business partners have a way of ignoring those things I've found. 

Totally agree re: devops.  As you note, some think devops is 

git commit -b prod
git push

[RPM 26,328]

Our shit started going down around 10:30. By 10:45 I had Leadership on 2 lines and they were both on different conference calls. On the bright side, all my shit was fubar so all I had to do was sit there and answer the phone. I already moved people to manually do what is normally done remote. Impressed the client, so I got that going for me.

[randomhorn 1,765]

39 minutes ago, DanTheHorn said:

No, the cybersecurity firm itself should have performed the test on their own before pushing the patch.

"worked fine on my box"

[0xdeadbeef 1,651]

2 minutes ago, Blotto said:

Thats what I don't get. If the  updates fucks all Windows boxes once installed, how does this make it out of Crowdstrikes QA? I assume they QA this shit before release, but perhaps i'm naive about that. 

I've seen lots of cases over the years where after  the testing was done, a software guy decided to slip in more more change, or make some should-not-change-the-behavior type change like fixing a typo in a log message or fix the indentation in the code. Since the change shouldn't impact the final product, they manually bypass the testing process and push straight to production.   Sooner or later that type change accidently deletes a line it shouldn't or changes a 1 to a 0 and breaks shit.  It's very likely is going to turn out to be a manual override of the standard process.     

 

And as someone said above, QA is lower caste. If a senior developer really wants to override process, in most places, they can. 
  

[Pato del Muerto 52,535]

Does crowd strike not use their own product, and push it to their devices first as a last qc?

[elfenix 34,331]

36 minutes ago, PvilleStang said:

The problem with that is how homogenous their testing machines can be.  Same problem Microsoft has.  The patches perform well in a bubble, but throw other software, XDRs, etc onto the test environment, and stuff crashes and burns.

Bring back the USB cart of death

 

[kevwun 15,811]

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

[Firemans4Horn 18,118]

My work email works. Everything else is FUBAR’ed

[Blotto 23,560]

2 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah it doesn't seem to be some esoteric fringe use case. It seems like the kind of thing that absolutely should have been  caught. 

[Captainant 27,690]

4 minutes ago, kevwun said:

Isn't the update breaking VMs also?  I don't see how it didn't get caught because it seems to break everything it's installed on.

Yeah my virtual desktop is hosed lol. Thankfully I have local admin on my work laptop and didn't auto pull the update so I'm not completely hosed. 

Our outlook servers, VPN, and a few other core shared services are completely børked though. 

Gonna be an interesting Friday lol

[crash_davis 23,114]

2 hours ago, HouTex said:

Using your personal machine to login to your company’s network is one work around that will work for many people.  Works for my law firm.  

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

1 hour ago, DanTheHorn said:

Yes but making sure you don't get a BOSD on reboot seems like a basic test before deploying a patch no matter how frequent patches are issued.

 

16 minutes ago, Pato del Muerto said:

Does crowd strike not use their own product, and push it to their devices first as a last qc?

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Edited July 19 by crash_davis

[immamac 42,877]

I'm not sure how there isn't a mass customer exodus after this. The amount of revenue impact this event is going to have should put crowd strike completely out of business. 

[ztejas 43,524]

Yeah - Xbox app/store was down for me last night on my PC. 

Didn't realize it was connected to a larger issue.

Woof.

[kevwun 15,811]

You would think this would seriously harm them.  Not like it's something you can deal with like causing printing problems or making another program to not work.  It's bricked every computer it's used on.

Edited July 19 by kevwun

[DanTheHorn 4,938]

1 minute ago, crash_davis said:

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Just saw interview with Crowdstrike CEO saying it did not affect all Windows versions or patch levels. So it sucks to be Crowdstrike this morning.

[Dahobbs 17,627]

Had a flight to Tahiti today for a cruise leaving tomorrow at 3 pm. Been scrambling since 5 am to try and find a way to make it to Tahiti in time as everything is prepaid and non refundable at this point. I may have sort have found a solution by rebooking on several different airlines that were not affected. Unfortunately that solution costs an extra 6k and has me flying all over for the next 24 hours. 

Side note, walking in to IAH and seeing all the computer screens with the BSOD convinced me to abandon any attempt of solving this issue through United. Now waiting around at Hobby. 

[kevwun 15,811]

Maybe they shouldn't be testing on Windows 10 version 1803.

[Clintonaldo 2,576]

LOL NERDS!

[Dahobbs 17,627]

6 minutes ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

Just is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

[MissingInAction 25,132]

35 minutes ago, boilerhorn said:

QA is also considered a lower caste in the IT/dev world.  Prod deployment QA is even lower than dev QA.

Can confirm./notatester

[RPM 26,328]

5 minutes ago, immamac said:

I'm not sure how there isn't a mass customer exodus after this. The amount of revenue impact this event is going to have should put crowd strike completely out of business. 

I work for one of the Big 4. Trust me they are NOT fucking happy.

[PvilleStang 2,917]

VMs are the easy ones to repair, it's the end user laptops that are being a PITA right now.  Forcing safe mode when the end user is remote and doesn't understand how to use F keys is almost as fun as trying to show aggy their empty trophy case.

[crash_davis 23,114]

Just now, Dahobbs said:

What, you don't test by immediately pushing an update to production for all of your worldwide customers simultaneously? Chicken. 

Well, let's start with these degenerate asshats pushing code on a Friday. Think of all the IT people who are going to have to work all weekend. Those same IT people will be sure to start looking at competitor software this weekend as they are working. Cloudstrike done fuck themselves good.