Jump to content

Shit's Borked

RPM

By RPM
in Please do this better?

 Share

Recommended Posts

  • Replies 6.3k
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

immamac
immamac

Tornado hit data center. Sorry for downtime. Going to have more downtime to fix later Please report weird shit here but I'm with my family and mobile only. I just spent about an hour on a bench o

hookemATL
hookemATL

I had to draw porno. Wasn’t great but I finished

immamac
immamac

https://www.surlyhorns.com/board/settings/account-security/#piiDataRequest if you want to know what data could have possibly been leaked request here.  The passwords are encrypted at rest in

Posted Images

blacklab

blacklab 11998

Posted
Posted
1 hour ago, Vertigo said:

Yep, cert just expired. Honest question but why don’t you guys host the site on a PaaS? Seems like so many of these types of issues are auto managed for you on a platform like Heroku or Azure. 

Because it would turn the site from profitable to a money loser

1 hour ago, Captainant said:

ffs guy, use a certificate manager or some shit to update your certs automatically. Use AWS ACM you wouldn't have this issue anymore and the SSL certs are free. Don't need anything on AWS to use it, not even domain registration on route53 

EDIT: Huh, their SSL provider, Let's Encrypt, says they do automatic cert rotation. No idea why it's broken, but cmon man

The automation works fine for a single server site . When you have multiple servers running it's not just a one liner.

To renew the cert I had to:

  • change nginx config
  • turn off haproxy
  • restart nginx
  • run certbot script
  • cat keys together and move to proper location
  • change nginx config
  • restart nginx
  • start up haproxy

Yeah, I could write a script to do this but I'm fucking lazy

 

Link to comment
Share on other sites
Captainant

Captainant 27989

Posted
Posted
34 minutes ago, blacklab said:

The automation works fine for a single server site . When you have multiple servers running it's not just a one liner.

To renew the cert I had to:

  • change nginx config
  • turn off haproxy
  • restart nginx
  • run certbot script
  • cat keys together and move to proper location
  • change nginx config
  • restart nginx
  • start up haproxy

Yeah, I could write a script to do this but I'm fucking lazy

 

Or you could stick the whole site behind a load balancer and then only have to worry about certing that single point instead of every backend server. Shit, are we load balancing at the DNS level instead of routing to a load balancer box?

Link to comment
Share on other sites
immamac

immamac 43584

Posted
Posted
29 minutes ago, blacklab said:

haproxy is our load balancer, it does take care of the https, the hosting servers do not do https

haproxy is running on a box that is also running nginx for hosting, no point in having that server sit at 1% usage

Yeah we terminate at the LB level. LB was misconfigured and didn't acknowledge and update the certs so shit got booked. 

Won't happen again. Please give me your neg rep. 

Link to comment
Share on other sites
  • 2 weeks later...
  • 2 weeks later...
Celery Man

Celery Man 21028

Posted
Posted

Just as a datum - I was having problems writing replies in Chrome last night.  The reply box was very short - clicking in it would drop it down enough for the "insert other media" box and maybe the submit reply box, but no space for the body/would not place a cursor.  I manually updated Chrome (which should be set to autoupdate, but... meh) and it functions normally now.  Other websites seemed fine, Firefox was working normally, based on the lack of anyone else saying anything about the same problem it's probably just something wonky in Chrome... or actually it may have been related to a flash update that I'm now remembering installed last night.  Fuckin adobe.

Link to comment
Share on other sites
sidis

sidis 9857

Posted
Posted
9 hours ago, blacklab said:

are you on a browser or tapatalk? If a browser which browser?

There are no ads on the site for people who are signed in.

 

2 hours ago, immamac said:

Strange there is no mechanism to show ads to anyone that is signed in and we don't run "hijack" ads. You should consider doing a virus/malware scan. 

just sent you both a pm with a screen grab video of what i am talking about.

Link to comment
Share on other sites
solamente73

solamente73 1132

Posted
Posted (edited)

iPad Pro running iOS 12.1 Safari, private browsing:  hijack ads when trying to sign in at https://www.surlyhorns.com/board/index.php 

Happened a couple of times this morning.  This time I was able to sign in before getting hijacked.

Edited by solamente73
Sign in, log in, 220, 221....
Link to comment
Share on other sites
  • 5 weeks later...
immamac

immamac 43584

Posted
Posted

Just an update for those curious about the downtime this morning that was unplanned:

Our main database server basically shit itself this morning and is effectively dead. Catastrophic failure, luckily we didn't even have to restore from backup and we lost no data from just failing over to our slave database host. Unfortunately I had the management software set to manual instead of automatic otherwise there would have been no disruption in service. The 503 errors that were happening afterwards were just due to the server still being in our load balancing rotation - I fixed that and everything should be good to go. No worries we have another slave and off site backups in the event of another failure on the current main database.

Link to comment
Share on other sites
RPM

RPM 26431

Posted
  • Author
Posted

I may have spotted what's causing the double post problem. If you reply to the thread before the last page it looks unresponsive. But open a new tab and check the thread it has the post. People keep clicking until they get a response or give up.

Link to comment
Share on other sites
  • 4 weeks later...

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing


Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business and Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Subscribe!... Donate!... Advertise... COOKIE MONSTER!

×
×
  • Create New...