Jump to content

Recommended Posts

Posted

Alright. I fucked up one minor otherwise would have had a flawless victory today at the datacenter. Less than 3 minutes of downtime and I power cycled every server at least once with no hiccups except this search thing and the main proxy going down at the same time as the backup coming up. 

Posted
On 8/17/2023 at 8:29 AM, TonyTexas said:

Frequently, when I hit Surly Horns on the top banner to go to back to topic listings, this is what pops up. NBD but I thought I’d make you aware. BTW, this is on Chrome on an iPhone. 

 

IMG_1412.png

You should charge your phone. 

14 hours ago, Longhorn_Fan68 said:

hurry up! I can't post post bullshit memes!

Close to home. 

  • Haha 1
Posted

Does the activity feed seem faster on load to the heavy users of it? Generally speaking I think the site is overall more responsive, but want to get some other folks to chime in on it.

Posted
36 minutes ago, shadow_operative2.0 said:

imma, you’re gonna get mad, call me a bunch of names, but you probably kinda want to figure this shit out before CFB starts in 2 days. 

Yeah im pretty pissed off about this one. No idea what is causing this it seemed to be working well for this week then poof index drops out of nowhere. 

  • Rage+1 1
Posted
40 minutes ago, thunderlounge said:

Cron job to reindex nightly at midnight?

Band-aid, I know. Just spit ballin. 

I did that before as a bandaid and I stopped and can't remember why. It was causing some fuckery. I've got the support team for the forum software looking into it. 

Posted
9 minutes ago, thunderlounge said:

That's wild. I hate to see it be an issue, but at the same time on the other side of the coin, my curiosity as to the underlying cause is high.

 

I thought it was a Rando doing it to fuck with us before because elastic was unsecured and open to the world if you knew the url. I locked that down via firewall rules now, so only the nodes themselves and the web nodes have access. The forum software people say there isn't a way to do this in their software so it must be something else. 

Posted

Fascinating and infuriating all in the same breath. Wow.

Maybe it’s not able to keep up with updates, and the queue gets too long after so long, and thus pukes?

Not that you haven’t thought of that, but I’m not familiar with how elastic reacts under high data, high traffic situations. High data, yes, but not both.

You’d think it would be able to keep pace, maybe not. 

Posted
26 minutes ago, thunderlounge said:

Fascinating and infuriating all in the same breath. Wow.

Maybe it’s not able to keep up with updates, and the queue gets too long after so long, and thus pukes?

Not that you haven’t thought of that, but I’m not familiar with how elastic reacts under high data, high traffic situations. High data, yes, but not both.

You’d think it would be able to keep pace, maybe not. 

it's dropping everything at like 3AM - I did notice that the new server I added into the rotation did have public access so I revoked that and maybe that'll do the trick.

Posted

Fuck. Sorry guys. I've locked this bitch down and it's still just fucking up. Gonna do one more thing then switch to mysql which will be a huge performance hit but won't fucking blow up ever. 

  • Like 1
Posted
25 minutes ago, immamac said:

Elasticsearch for search.

Wtf is up with the index??? I've done some massive enterprise scale ES cluster sharding work (>20TB total index live data) and usually these sorts of issues come with massive OOM or IOwait metrics, or there's some less than optimal sharding going on. 

What version ES are you running? Hopefully it's 6.X or newer - honestly should move to open search to avoid licensing costs anyways 

Posted
34 minutes ago, Captainant said:

Wtf is up with the index??? I've done some massive enterprise scale ES cluster sharding work (>20TB total index live data) and usually these sorts of issues come with massive OOM or IOwait metrics, or there's some less than optimal sharding going on. 

What version ES are you running? Hopefully it's 6.X or newer - honestly should move to open search to avoid licensing costs anyways 

That was the first teardown and rebuild. Invision doesn't support api for ES > 7.10. Built ES from source. Has way more run room than needed for mem (entire index in memory + 20%) it's something dropping the index. If you know of a query analyzer or audit I will run it. Its pissing me the fuck off. 

Posted

I feel like it's a stupid question but... Is anything in the invision orchestration software calling the reindex API, or anything else that could be triggering a refresh? Maybe some sort of thrashing condition from quorum for the cluster shifting between nodes too frequently?

Tough to speculate more without knowing master and data node counts, data volume, and shard distribution across the nodes

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.



×
×
  • Create New...