Shit's Borked

[Braff Zacklin 13,626]

42 minutes ago, immamac said:

Chicago latency from Texas is lowest and geographically far enough to where the same event is unlikely to have effected both. 

Florida has too many hurricanes. North Carolina didn't have any capacity, didn't wanna do Arizona, but could look into it. California or any west region is too high latency for me same with the full on northeast regions. 

Surly is on the backbone, which is one of the reasons we don't have to have a cdn to serve content surly gets to the open web in 1 hop, through the redundant interconnect switch. Surly for users in Texas is actually faster than the majority of CDNs. For users outside of Texas it's likely just as fast. When I travel I test it pretty thoroughly and the latency and bandwidth is totally fine. 

I was only considering locations on backbone and not buried behind 2-3 interconnects to get to there. 

I may set up a data replication service off backbone to make DR trivial, the hard part right now is dealing with data syncing between sites as replication is one way by design right now. 

This was a catastrophic event and all system durability was proven pretty well which i am happy about. There has never been a time in this current Datacenter where we have had complete power loss. We had a network hiccup a few years ago due to a misconfigured switch but that was a blip in comparison to today's real deal full dark outage. 

You're good, dude. My post wasn't intended as criticism of you or this site. I just wanted to diss Chicago.

Edited May 26 by Braff Zacklin
wasn't

[Lurch 5,931]

For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 

@terriblemaps

[Rimbo 25,985]

One of the sites I like (killsixbilliondemons.com) is hosted by NixiHost, whose servers are in the DFW area. They were also out this morning, but are back up now.

[BurntEyes 6,003]

57 minutes ago, immamac said:

For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 

Why do you hate South Dakota and New Mexico. The Surly population in those meccas of assholes is massive. 

[Gatorubet 30,247]

1 hour ago, immamac said:

I was only considering locations on backbone and not buried behind 2-3 interconnects to get to there. 

I may set up a data replication service off backbone to make DR trivial, the hard part right now is dealing with data syncing between sites as replication is one way by design right now. 

[Al Bundy's Napoleon Hand 13,585]

Borkage de-intensifying

 

[Bozo_Casanova 26,495]

7 hours ago, immamac said:

Tornado hit data center. Sorry for downtime.

Loser talk

[Gut Wagon 8,083]

Aggy cloud seeding finally pays off for them in their holy war against sippy. 

[Al Bundy's Napoleon Hand 13,585]

Borkage currently intesifying

 

[Hate 20,116]

5 hours ago, Hornius Emeritus said:

Don't worry about it. Shit happens. You don't owe us anything.

However, I'm about to get evil on those assholes in the 2025 recruiting thread who insist on discussing whether Katy is in Houston. Jeeze.

Of course Katy is Houston. 

[Davis Lane 5,587]

5 hours ago, immamac said:

For non nerd version. I only put equipment I own on the red dots which are very specific facilities. Choosing to do it this way makes it cost effective and the most robust and reliable deployment possible. 

[Hagbard Celine 10,875]

screw fixing search

it's so bad, just turn it off

kill it with fire

is there a way you can macgyver the search box to register as an ad impression and redirect to google?

[ZanTheFan 49]

10 hours ago, immamac said:

I was trying to get shit back up with screaming children doing the dino exhibit at Lady bird wildflower center. 

Fucking legend. 

[Pato del Muerto 52,529]

@immamac not sure if you know or care for any metrics for revenue, but it looks like view count isn’t working at leas Ron threads started today. They all show 0 views even though they have replies. 

[immamac 42,869]

8 hours ago, Pato del Muerto said:

@immamac not sure if you know or care for any metrics for revenue, but it looks like view count isn’t working at leas Ron threads started today. They all show 0 views even though they have replies. 

I'm aware I'm flipping things back around and it'll fix probably later today. 

[SquishMitten 10,417]

Was Texassports.com being all borked yesterday caused by the same issue? It was a really weird feeling when two of my most accessed sites were down. And I’m currently in the middle of The Last Stand book series which is about an EMP blast, so my mind went to some weird places for a bit.

[Beau Vine 54,261]

Edited May 27 by Beau Vine

[immamac 42,869]

44 minutes ago, SquishMitten said:

Was Texassports.com being all borked yesterday caused by the same issue? It was a really weird feeling when two of my most accessed sites were down. And I’m currently in the middle of The Last Stand book series which is about an EMP blast, so my mind went to some weird places for a bit.

no idea, but yes this took a lot of sites down it's not some shitty little third rate colo.

[PvilleStang 2,913]

On 5/26/2024 at 11:22 AM, Buzzrock said:

You should try the cloud. Clouds don’t get hit by tornadoes. 

 

On 5/26/2024 at 11:29 AM, Gatorubet said:

The cloud is just someone elses' computers.

FIFY.

But, @immamac, have you considered going active / active, and doubling up the A records in DNS so you can distribute traffic accordingly?  Or would the sync times between the two DCs just kill that idea?

[immamac 42,869]

28 minutes ago, PvilleStang said:

 

FIFY.

But, @immamac, have you considered going active / active, and doubling up the A records in DNS so you can distribute traffic accordingly?  Or would the sync times between the two DCs just kill that idea?

Why? It introduces so much complexity for basically no gain. 

 

If surly produced millions a year in revenue and did hundreds of financial transactions a second I'd do some of this stuff, it's just not really worth it or feasible for the level of revenue. 

[Captainant 27,682]

I didn't realize surly was rawdogging it to the internet with no CDN, that definitely makes the fail over and fail back much more complex. But on the other hand, if you rarely ever use the DR resources because of the hassle to change over, what's the point in maintaining it? Why not just do change data capture for the posts to the off-site and rebuild when needed?

 

[immamac 42,869]

34 minutes ago, Captainant said:

I didn't realize surly was rawdogging it to the internet with no CDN, that definitely makes the fail over and fail back much more complex. But on the other hand, if you rarely ever use the DR resources because of the hassle to change over, what's the point in maintaining it? Why not just do change data capture for the posts to the off-site and rebuild when needed?

 

Why do that when you can just replicate off site? DR site is exactly that. A site that can scale up in the event of DR to serve traffic, it still has to have a complete copy of the data there. The data hotel I was talking about setting up as a tertiary never to serve up anything place is what you are suggesting, which isn't a bad idea I just don't know if it fucking matters. 

The reality is it wouldn't have been worth flipping to DR then having to wait till night to take the downtime and switch back to main and have circular replication or an extra redirect for another day because DNS propagation is potato even with 60s TTL. I'd rather just take the downtime up front if it's only an hour or 2 than to spend 12-16 hours of my own time making sure shit goes well and is fine.

Posts are only one part of the whole thing that makes the site go. 

Check this out. I had to use a Dallas DB server that I replicate to just in case some shit happens in the rack to serve traffic for the day. This is what that looks like. Sustained 300+Mbps and averaging well into the 400+Mbps range. Just for the database. I think people fail to recognize how big the site is constantly.

 

 

[Captainant 27,682]

1 hour ago, immamac said:

Why do that when you can just replicate off site? DR site is exactly that. A site that can scale up in the event of DR to serve traffic, it still has to have a complete copy of the data there. The data hotel I was talking about setting up as a tertiary never to serve up anything place is what you are suggesting, which isn't a bad idea I just don't know if it fucking matters. 

The reality is it wouldn't have been worth flipping to DR then having to wait till night to take the downtime and switch back to main and have circular replication or an extra redirect for another day because DNS propagation is potato even with 60s TTL. I'd rather just take the downtime up front if it's only an hour or 2 than to spend 12-16 hours of my own time making sure shit goes well and is fine.

Posts are only one part of the whole thing that makes the site go. 

Check this out. I had to use a Dallas DB server that I replicate to just in case some shit happens in the rack to serve traffic for the day. This is what that looks like. Sustained 300+Mbps and averaging well into the 400+Mbps range. Just for the database. I think people fail to recognize how big the site is constantly.

 

 

Is that 300-500Mbps for just the DB server? Or is that the front door app server? 

Sorry for all the probing questions lol, I have a customer that puts a ton of work into their DR but almost never fails over because of DNS headaches as you describe. They're trying to decide if it's really worth the power and man hours of maintaining a copy in a DR location given they've never actually used it in the last decade lol. 

It's crazy that the persistence isn't maintained in the DB or ES layer, where you could more easily keep a synchronized copy in a cluster config. I don't know the ins and outs of the board software, but it seems surprisingly stateful and brittle in that a new instance of the forum couldn't just read/write to the same data layer 

[PvilleStang 2,913]

Simple script could automate the DNS failover, then manually fail it back.  

I also think you should mirror almost everything to the DR site, but replace all embedded pics with Billy_fellating_a_Thujone_donger.gif.

 

In all seriousness, what is the size of Surly in its entirety on disk, and do you tier the data storage, or is it all on one storage device?

[immamac 42,869]

5 minutes ago, PvilleStang said:

Simple script could automate the DNS failover, then manually fail it back.  

I also think you should mirror almost everything to the DR site, but replace all embedded pics with Billy_fellating_a_Thujone_donger.gif.

 

In all seriousness, what is the size of Surly in its entirety on disk, and do you tier the data storage, or is it all on one storage device?

It's on an array and each web node has local nvme cache serving it up for the internal cdn (media.surlyhorns.com). So it's tiered for access with cache misses getting pulled from a large disk array with plenty of dram cache and also replicated in real time (block) and every 15m to DR on the media side. It's measured in high single digit TB, but I'd say the hot data is in the 200-300GB range. 

The databases run on all flash. 

The problem isn't the mechanism of fail over. That's all automated. It's DNS propagation paired with intentional one way replication for active/passive. I have active/active intra rack. Database failovers that are planned are sub 10ms and unplanned I have not fully automated because I'd rather take downtime than risk split brain which is a bitch to remediate. 

Uptime isn't a priority because we aren't pumping revenue every second on this site. Data is the priority. 

[TexasHooch 5,122]

My "show reply" button doesn't work anymore on desktop.  I have to refresh the screen.

[thunderlounge 7,871]

Worked ok for me a few minutes ago. 

[TexasHooch 5,122]

23 minutes ago, thunderlounge said:

Worked ok for me a few minutes ago. 

I think it's related to new banner ads that are appearing at the bottom.  Being Burnt Ends, I'm guessing you don't get those.  Perhaps it's time to jump back in.

[TheStoicPaisano 9,922]

YouTube embeds requiring captchas now?

Edited June 15 by TheStoicPaisano

[immamac 42,869]

16 minutes ago, TheStoicPaisano said:

YouTube embeds requiring captchas now?

Every time or just sometimes? 

[thunderlounge 7,871]

Can’t say I’ve seen one of those. 

[TheStoicPaisano 9,922]

14 minutes ago, immamac said:

Every time or just sometimes? 

Just for the last 24 hours (at most, prob more like 12 hours). In previously made posts and my attempt to post. 

The Mike Leach thread has a lot of examples.

Edited June 15 by TheStoicPaisano

[immamac 42,869]

Just now, TheStoicPaisano said:

Just for the last 24 hours (at most, prob more like 12 hours). In previously made posts and my attempt to post. 

Maybe you are getting rate limited? 

[TheStoicPaisano 9,922]

All better now, whatever it was.

[ConferenceRoom 6,144]

Test. 

Posts are super slow to save this morning. 

Edited June 25 by ConferenceRoom

[ConferenceRoom 6,144]

Test. 

[thunderlounge 7,871]

Concur. It’s posting slow. 
 

Also noticing streams are getting wonky. 
 

For example, a new post in the LOL thread was in my feed, but showed the post right before the new one, and looked like a visited link instead of a new one.

So if there were 10 posts, marked read, along comes post #11. It triggers the feed, but shows post #10 instead of #11, and appears read instead of unread. 

[Js1 85,437]

We haven't even hired a new baseball coach and it's already broken

[immamac 42,869]

Looking into it. 

[immamac 42,869]

One of the search nodes borked overnight just started it up again and we are good. 

[Captain Ron 12,883]

BTW - this slow posting thing appears to be happening again tonight.

[immamac 42,869]

8 minutes ago, Captain Ron said:

BTW - this slow posting thing appears to be happening again tonight.

Fixed. 

[markstanco 12,581]

Lots of double and triple posting tonight. 

[immamac 42,869]

29 minutes ago, markstanco said:

Lots of double and triple posting tonight. 

yeah the secondary web node doubles as an elasticsearch node and the high traffic caused some memory issues that I resolved. Shouldn't be an ongoing issue and set up an alert to handle.

[thunderlounge 7,871]

8 hours ago, immamac said:

yeah the secondary web node doubles as an elasticsearch node and the high traffic caused some memory issues that I resolved. Shouldn't be an ongoing issue and set up an alert to handle.

Am not! Now, anyway. 

[shadow_operative2.0 16,006]

Site is pretty damn slow today, even more than usual on pages with lots of pics/embeds. 

 

web01 currently 

[immamac 42,869]

Hurricane tubes lol I'll look into some metrics but everything seems operational without issue

[shadow_operative2.0 16,006]

Goddamn inchworms! 

[Rimbo 25,985]

I'm looking forward to the incident response report on THIS one.

[Js1 85,437]

What

the

fuck