Crowdstrike Outage a/k/a Everything's Fucked

[kevwun 15,758]

Just now, PvilleStang said:

VMs are the easy ones to repair, it's the end user laptops that are being a PITA right now.  Forcing safe mode when the end user is remote and doesn't understand how to use F keys is almost as fun as trying to show aggy their empty trophy case.

Not to mention that like every fucking laptop seems to have a different key combination to start safe mode.

[TonyTexas 10,504]

Lulz

 

[Dahobbs 17,521]

14 minutes ago, crash_davis said:

You're the reason why your company will soon be held ransom by some Russian hacker gang.

 

 

This is just bad process governance. Not sure how this could have been missed if they have the proper test cases and if they internally test for a period of time before releasing in Prod. Not sure how a 83 BILLION dollar company is this inept?

Or if they had just done a staged rollout of the update, this would have been identified before it shut down the fucking planet. Fucking idiots. 

[kevwun 15,758]

The sphere blue screen is real, but it's from February.

[HOOKEM4 5,036]

You better edit that fucking twitter post. Jesus dude.

[PvilleStang 2,875]

6 minutes ago, crash_davis said:

Well, let's start with these degenerate asshats pushing code on a Friday. Think of all the IT people who are going to have to work all weekend. Those same IT people will be sure to start looking at competitor software this weekend as they are working. Cloudstrike done fuck themselves good.

 

This was a Thursday evening patch release.  And frankly, it's better to do it then vs Friday at 5pm and walk out the door.

[Js1 85,258]

@MissingInAction wtf. You basically broke this on mobile with that post 

[Kel Varnsen 8,458]

Good thing I chose SentinelOne as my EDR.

[DanTheHorn 4,913]

4 minutes ago, MissingInAction said:

Not sure if real but still funny

No it is not funny, do you have the right thread?

[SubliminalHorn 6,780]

Got on a nonstop off standby. Take off soon. Thank baby vishnu 

[MissingInAction 25,002]

1 minute ago, Js1 said:

@MissingInAction wtf. You basically broke this on mobile with that post 

Sorry. I don't have a Twitter account so the embed fucked up

[kevwun 15,758]

Yeah, we noticed when you posted all of French twitter.

[MissingInAction 25,002]

1 minute ago, DanTheHorn said:

No it is not funny, do you have the right thread?

Sorry I fucked up. Should be gone now.

[crash_davis 23,096]

3 minutes ago, PvilleStang said:

This was a Thursday evening patch release.  And frankly, it's better to do it then vs Friday at 5pm and walk out the door.

Same thing. No patches after Thu COB. It's asking a lot to unfuck shit one day before the weekend. Sun nigh thru Wed night/Thu morning was my company's window.

[MissingInAction 25,002]

Can we get a dev version of surly?

[Deej 76,750]

Frost says they can't accept deposits right now. Something tells me they'll find a way to withdraw from my accounts. 

[Chet Steadman 940]

MIA just CrowdStrike’d this thread with his post

[Gatorubet 30,183]

American did not send me a single text about my flight being canceled prior to my arrival at the airport.  I am supposed to be at AAJ convention in Nashville today, but they gave me a 16 hour layover in Charlotte and a flight out to Nashville Saturday morning.  

The good news is, I can swill free alcohol at this Priority Pass lounge. First world problems, I guess

i’m on a few waitlists on other airlines so fingers crossed and another old-fashioned on the way

Edit:  here is the last text from American

Edited July 19 by Gatorubet

[Dahobbs 17,521]

Are the systems with bitlocker extra turbo fucked? 

Edited July 19 by Dahobbs

[Bozo_Casanova 26,429]

 

[immamac 42,832]

1 minute ago, Dahobbs said:

Are the systems with bitlocker extra turbo fucked? 

No. 

[Captainant 27,620]

Just now, Dahobbs said:

Are the systems with bitlocker installed extra turbo fucked? 

Oh we got hit by that! We have to call our central IT, verify our identity, read the serial number sticker off our machine, and then boot into safe mode, enter the bitlocker pin, and then go into command line and delete the bad DLL. Reboot and you're good!

Super fast and easy and scalable lol

[RPM 26,315]

8 minutes ago, MissingInAction said:

Sorry I fucked up. Should be gone now.

Do you work at Crowdstrike?

[Captainant 27,620]

Oh and the Mercedes F1 team is sponsored by crowdstrike lol

 

[Dahobbs 17,521]

Just now, RPM said:

Do you work at Crowdstrike?

At least he fixed the problem in 5 minutes. 

[Js1 85,258]

Couldn’t scan my barcode at the gym and had to sign in manually, with pen and paper

[kevwun 15,758]

I thought about Mercedes.  And there just happens to be a race this weekend.

Edited July 19 by kevwun

[Dahobbs 17,521]

2 minutes ago, Captainant said:

Oh we got hit by that! We have to call our central IT, verify our identity, read the serial number sticker off our machine, and then boot into safe mode, enter the bitlocker pin, and then go into command line and delete the bad DLL. Reboot and you're good!

Super fast and easy and scalable lol

Yeah, I figured it might be an issue. End users are already going to struggle to deploy this fix. Adding bitlocker on top of it seems like a nightmare. And that's assuming central IT can actually access those keys. 

[crash_davis 23,096]

Good news! My go to pron sites still work. Companies should seek IT admins from pron companies. Those sites never go down. Imagine how much war humanity would wage if we didn't have access to fine bitches.

Edited July 19 by crash_davis

[cactusflinthead 27,598]

 

[Shaggy3.0 1,695]

2 minutes ago, RPM said:

Do you work at Crowdstrike?

even if he did, would he admit it?  would anyone list CS on their resume after this?

 

[MissingInAction 25,002]

4 minutes ago, RPM said:

Do you work at Crowdstrike?

[SubliminalHorn 6,780]

9 minutes ago, Gatorubet said:

American did not send me a single text about my flight being canceled prior to my arrival at the airport.  I am supposed to be at AAJ convention in Nashville today, but they gave me a 16 hour layover in Charlotte and a flight out to Nashville Saturday morning.  

The good news is, I can swill free alcohol at this Priority Pass lounge. First world problems, I guess

i’m on a few waitlists on other airlines so fingers crossed and another old-fashioned on the way

If I had known you were at CLT I would have given you a good shoulder check 

[Dahobbs 17,521]

2 minutes ago, Shaggy3.0 said:

even if he did, would he admit it?  would anyone list CS on their resume after this?

 

Anarchist? And really, anyone from the sales team can point out that they got everyone in the fucking world to buy their shitty product/service. 

[MissingInAction 25,002]

3 minutes ago, Dahobbs said:

Yeah, I figured it might be an issue. End users are already going to struggle to deploy this fix. Adding bitlocker on top of it seems like a nightmare. And that's assuming central IT can actually access those keys. 

The keys should be saved in active directory.

[boilerhorn 1,371]

10 minutes ago, Dahobbs said:

Are the systems with bitlocker extra turbo fucked? 

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

[Gatorubet 30,183]

3 minutes ago, SubliminalHorn said:

If I had known you were at CLT I would have given you a good shoulder check 

Still chillin’ in the big easy.  PM lobo - he probably knows some guys

[randomhorn 1,722]

18 minutes ago, Deej said:

Frost says they can't accept deposits right now. Something tells me they'll find a way to withdraw from my accounts. 

The withdrawal system is an indestructible AS/400 kept secure at area 51. The deposit system is a fragile leftover box of parts running windows 95.

[Celery Man 20,929]

30 minutes ago, PvilleStang said:

VMs are the easy ones to repair, it's the end user laptops that are being a PITA right now.  Forcing safe mode when the end user is remote and doesn't understand how to use F keys is almost as fun as trying to show aggy their empty trophy case.

i'm in a discord giving shit to some old work buddies and one of them is complaining that the process to get a vm into safe mode in azure is actually complicated. of course he was the person most likely to cause a global outage so I'm not sure about it.

 

if i was responsible for this i think i'd move and change my name

[MissingInAction 25,002]

5 minutes ago, MissingInAction said:

The keys should be saved in active directory.

When I worked desktop support part of our image script would save the keys under the computer name in the security tab in AD.

[chainsaw 9,850]

NCAA Football = King Tut

[Gatorubet 30,183]

1 hour ago, boilerhorn said:

True agile has that in tier 7 of the testing, which is customer-testing.  Windows boot, post-deployment, will now be part of the "DOA Testing" adding 5 minutes to the CI, slowing developers.

9 minutes ago, boilerhorn said:

They are not.  Just need the keys to get through to the command prompt.  That said, you either had to save your keys from before or access them from your corp portal.  The process is quite easy for those who are tech savvy.  Less so for those who are not.

 

[89Horn 1,512]

[Blotto 23,485]

Lulz, look at this fucking dildo. He's been getting into his kid's hair gel again. From the looks of it, he might be drinking it. 

Saw this  earlier as well..

Quote

Also the CEO was the CTO of McAfee in 2010 when they released an updated that made the antivirus think svchost.exe (a system file) was a virus. Bricked tens of thousands of computers.

He's got the golden touch. 

[Celery Man 20,929]

[Brisketexan 105,043]

Just now, Blotto said:

Lulz, look at this fucking dildo. He's been getting into his kid's hair gel again. From the looks of it, he might be drinking it. 

Saw this  earlier as well..

He's got the golden touch. 

Hmmm....there's a real chance he'll be forced out as CEO.

After the board grants him a $150 million exit package.

[mdmost 42,997]

So this is what Y2K was supposed to be like?

[Celery Man 20,929]

Just now, Blotto said:

Lulz, look at this fucking dildo. He's been getting into his kid's hair gel again. From the looks of it, he might be drinking it. 

Saw this  earlier as well..

He's got the golden touch. 

there's a specific brand of leadership dude I've seen several times at these kinds of companies that is.... gen x guy who traded the puka shell necklace for a print button down but has changed nothing else about his style since 1998

[Shaggy3.0 1,695]

uncle bob is probably smiling ear to ear.  he deserves it.

i met him at sfo in 2012 during confluent.  the guy is wise and an absolute wizard.

the longer this plays out, his pov becomes more important because he's right.  software engineering, as a discipline, should require higher credentials and licensing.  he took a lot of shit for his takes.  but i agree with him 10000%.  

https://dev.to/bosepchuk/is-uncle-bob-serious-dhi?ref=highscalability.com

  

[Captainant 27,620]

3 minutes ago, Blotto said:

man if "MBA" were a person lol...