Jump to content

Shit's Borked

RPM

By RPM
in Please do this better?

 Share

Recommended Posts

  • Replies 6.3k
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

immamac
immamac

Tornado hit data center. Sorry for downtime. Going to have more downtime to fix later Please report weird shit here but I'm with my family and mobile only. I just spent about an hour on a bench o

hookemATL
hookemATL

I had to draw porno. Wasn’t great but I finished

immamac
immamac

https://www.surlyhorns.com/board/settings/account-security/#piiDataRequest if you want to know what data could have possibly been leaked request here.  The passwords are encrypted at rest in

Posted Images

immamac

immamac 42839

Posted
Posted

Alright. I fucked up one minor otherwise would have had a flawless victory today at the datacenter. Less than 3 minutes of downtime and I power cycled every server at least once with no hiccups except this search thing and the main proxy going down at the same time as the backup coming up. 

Link to comment
Share on other sites
DDD Dad

DDD Dad 14332

Posted
Posted
On 8/17/2023 at 8:29 AM, TonyTexas said:

Frequently, when I hit Surly Horns on the top banner to go to back to topic listings, this is what pops up. NBD but I thought I’d make you aware. BTW, this is on Chrome on an iPhone. 

 

IMG_1412.png

You should charge your phone. 

14 hours ago, Longhorn_Fan68 said:

hurry up! I can't post post bullshit memes!

Close to home. 

  • Haha 1
Link to comment
Share on other sites
immamac

immamac 42839

Posted
Posted
36 minutes ago, shadow_operative2.0 said:

imma, you’re gonna get mad, call me a bunch of names, but you probably kinda want to figure this shit out before CFB starts in 2 days. 

Yeah im pretty pissed off about this one. No idea what is causing this it seemed to be working well for this week then poof index drops out of nowhere. 

  • Rage+1 1
Link to comment
Share on other sites
immamac

immamac 42839

Posted
Posted
40 minutes ago, thunderlounge said:

Cron job to reindex nightly at midnight?

Band-aid, I know. Just spit ballin. 

I did that before as a bandaid and I stopped and can't remember why. It was causing some fuckery. I've got the support team for the forum software looking into it. 

Link to comment
Share on other sites
immamac

immamac 42839

Posted
Posted
9 minutes ago, thunderlounge said:

That's wild. I hate to see it be an issue, but at the same time on the other side of the coin, my curiosity as to the underlying cause is high.

 

I thought it was a Rando doing it to fuck with us before because elastic was unsecured and open to the world if you knew the url. I locked that down via firewall rules now, so only the nodes themselves and the web nodes have access. The forum software people say there isn't a way to do this in their software so it must be something else. 

Link to comment
Share on other sites
thunderlounge

thunderlounge 7816

Posted
Posted

Fascinating and infuriating all in the same breath. Wow.

Maybe it’s not able to keep up with updates, and the queue gets too long after so long, and thus pukes?

Not that you haven’t thought of that, but I’m not familiar with how elastic reacts under high data, high traffic situations. High data, yes, but not both.

You’d think it would be able to keep pace, maybe not. 

Link to comment
Share on other sites
immamac

immamac 42839

Posted
Posted
26 minutes ago, thunderlounge said:

Fascinating and infuriating all in the same breath. Wow.

Maybe it’s not able to keep up with updates, and the queue gets too long after so long, and thus pukes?

Not that you haven’t thought of that, but I’m not familiar with how elastic reacts under high data, high traffic situations. High data, yes, but not both.

You’d think it would be able to keep pace, maybe not. 

it's dropping everything at like 3AM - I did notice that the new server I added into the rotation did have public access so I revoked that and maybe that'll do the trick.

Link to comment
Share on other sites
Captainant

Captainant 27654

Posted
Posted
25 minutes ago, immamac said:

Elasticsearch for search.

Wtf is up with the index??? I've done some massive enterprise scale ES cluster sharding work (>20TB total index live data) and usually these sorts of issues come with massive OOM or IOwait metrics, or there's some less than optimal sharding going on. 

What version ES are you running? Hopefully it's 6.X or newer - honestly should move to open search to avoid licensing costs anyways 

Link to comment
Share on other sites
immamac

immamac 42839

Posted
Posted
34 minutes ago, Captainant said:

Wtf is up with the index??? I've done some massive enterprise scale ES cluster sharding work (>20TB total index live data) and usually these sorts of issues come with massive OOM or IOwait metrics, or there's some less than optimal sharding going on. 

What version ES are you running? Hopefully it's 6.X or newer - honestly should move to open search to avoid licensing costs anyways 

That was the first teardown and rebuild. Invision doesn't support api for ES > 7.10. Built ES from source. Has way more run room than needed for mem (entire index in memory + 20%) it's something dropping the index. If you know of a query analyzer or audit I will run it. Its pissing me the fuck off. 

Link to comment
Share on other sites
Captainant

Captainant 27654

Posted
Posted

I feel like it's a stupid question but... Is anything in the invision orchestration software calling the reindex API, or anything else that could be triggering a refresh? Maybe some sort of thrashing condition from quorum for the cluster shifting between nodes too frequently?

Tough to speculate more without knowing master and data node counts, data volume, and shard distribution across the nodes

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing


Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business and Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Subscribe!... Donate!... Advertise... COOKIE MONSTER!

×
×
  • Create New...