Jump to content

Shit's Borked

RPM

By RPM
in Please do this better?

 Share

Recommended Posts

  • Replies 6.3k
  • Created
  • Last Reply

Top Posters In This Topic

Popular Days

Top Posters In This Topic

Popular Days

Popular Posts

immamac
immamac

Tornado hit data center. Sorry for downtime. Going to have more downtime to fix later Please report weird shit here but I'm with my family and mobile only. I just spent about an hour on a bench o

hookemATL
hookemATL

I had to draw porno. Wasn’t great but I finished

immamac
immamac

https://www.surlyhorns.com/board/settings/account-security/#piiDataRequest if you want to know what data could have possibly been leaked request here.  The passwords are encrypted at rest in

Posted Images

Rimbo

Rimbo 26028

Posted
Posted (edited)
On 2/16/2019 at 2:31 PM, immamac said:

We installed 4 new servers and there is going to be redundancy for stuff like this after today. Unless the whole rack goes down after this we shouldn't have any more hiccups. 

I think the real question here is -- who uses physical servers instead of AWS and other auto-scaling cloud solutions nowadays?

Edited by Rimbo
Link to comment
Share on other sites
immamac

immamac 42878

Posted
Posted
2 hours ago, Rimbo said:

I think the real question here is -- who uses physical servers instead of AWS and other auto-scaling cloud solutions nowadays?

Substantial amounts of people. So many so that AWS has it's own bare metal offerings as do all the other cloud providers. For us the downtime wouldn't be fixed by AWS or "auto-scaling" cloud solutions as it was the equivalent of an unplanned az outage which happens all the time in AWS/GCE/Azure anyone that uses physical hosts and has to do maintenance on them knows this. On the software side of things the reason we racked 4 more servers is to provide more redundancy and the ability for us to reduce our single points of failure that were not allowing for immediate auto-healing. 

Our cost is also substantially reduced by using our own infrastructure and managing it ourselves. The same compute/storage/bandwidth in amazon is roughly 5x the monthly cost of what we pay in the colo (with plenty of capacity to grow into). For instance we are about to start offering image storage for members, the bandwidth and storage alone to have the equivalent performance would far outweigh the cost of us just setting up our own SAN in the datacenter for a one time cost with no additional monthly cost.

During our "Off-peak" times we still have ~1000 concurrent users at any given time on the site, to have invision host it for us they were quoting close to 1700/mo (btw they host on their own infra) during our peak times that number is easily close to the 10k range which we were quoted closer to 6-7k/mo.

We are currently happy in the datacenter paying less than 700/mo combined for bandwidth, power, storage and rack space. We have over 240 cores of computer and nearly a TB of RAM (soon to have many TB of high performance SSD storage) That price doesn't change at all for the compute we have and makes no sense in the cloud.

 

Link to comment
Share on other sites
Rimbo

Rimbo 26028

Posted
Posted
1 hour ago, immamac said:

For us the downtime wouldn't be fixed by AWS or "auto-scaling" cloud solutions as it was the equivalent of an unplanned az outage which happens all the time in AWS/GCE/Azure

That's what multi-AZ load balancers are for. And multi-region deployments. And you can launch a new instance on different hardware, done right, with the push of a button.

 

1 hour ago, immamac said:

Our cost is also substantially reduced by using our own infrastructure and managing it ourselves.

Now that is a sensible reason. Although once you get into reserved instances and specialized pricing for high-capacity users, AWS' overall cost becomes a lot lower. RIs alone will trim 35% off your bill; when you have enough users that your downtime is 6000 requests/minute, that's when they call you up and say, "Hey, can we cut a deal with you?"

But then, there's another cost you're not talking about -- I assume you're doing this as volunteer work. If you were billing yourself for your hours at a legit hourly rate, what would the cost look like then, and how would that affect the total costs?

Link to comment
Share on other sites
Captainant

Captainant 27703

Posted
Posted

I agree, the site could probably be rebuilt from the ground up to use lambda, Aurora/Dynamo/whatever, and s3 to have a completely serverless architecture that would run for a fraction the cost - but shit man, it's working pretty well for now and I sure as shit don't wanna deal with running a fan forum for a bunch of assholes. So I'm more than happy with whatever our fearless leader decides to do with regards to running the site as long as it doesn't shit the bed too frequently lol. 

 

...

 

But for real, it prolly wouldn't be that much of a lift, holler if you feel like changing everything for marginal operational improvement lol

Link to comment
Share on other sites
immamac

immamac 42878

Posted
Posted
18 hours ago, Rimbo said:

That's what multi-AZ load balancers are for. And multi-region deployments. And you can launch a new instance on different hardware, done right, with the push of a button.

 

Now that is a sensible reason. Although once you get into reserved instances and specialized pricing for high-capacity users, AWS' overall cost becomes a lot lower. RIs alone will trim 35% off your bill; when you have enough users that your downtime is 6000 requests/minute, that's when they call you up and say, "Hey, can we cut a deal with you?"

But then, there's another cost you're not talking about -- I assume you're doing this as volunteer work. If you were billing yourself for your hours at a legit hourly rate, what would the cost look like then, and how would that affect the total costs?

 

12 hours ago, Captainant said:

I agree, the site could probably be rebuilt from the ground up to use lambda, Aurora/Dynamo/whatever, and s3 to have a completely serverless architecture that would run for a fraction the cost - but shit man, it's working pretty well for now and I sure as shit don't wanna deal with running a fan forum for a bunch of assholes. So I'm more than happy with whatever our fearless leader decides to do with regards to running the site as long as it doesn't shit the bed too frequently lol. 

 

...

 

But for real, it prolly wouldn't be that much of a lift, holler if you feel like changing everything for marginal operational improvement lol

Basically we got it running well and worked out some kinks in load balancing the php app (invision community) then didn't wanna mess with it during the rest of football season. Just so everyone knows it's running on HAProxy (LB now HA itself and auto health checking nginx endpoints) + Nginx (webserver) + PHP 7.2 + MySQL (standard HA implementation with manual failover ATM, automation incoming) + Redis (Caching) + Elastic Search (search only, fails to MySQL if ES not available). Getting this running on AWS's flavor of everything really just sounds like asking to open a pretty big can of worms since the software (which I cannot modify freely) isn't built to do it.

I finished configuration management (hiccup the other day) and blacklab recently fixed the automated renewal of the ssl cert. A lot of the hiccups were due to being lazy and/or just not having the hardware we wanted in the rack (we had it in our garages)

As I mentioned soon we will be adding a high performance storage array to the rack and that'll give us a path forward on some cool stuff like image/media hosting for members.

 

Link to comment
Share on other sites
Rimbo

Rimbo 26028

Posted
Posted
20 minutes ago, immamac said:

 

Basically we got it running well and worked out some kinks in load balancing the php app (invision community) then didn't wanna mess with it during the rest of football season. Just so everyone knows it's running on HAProxy (LB now HA itself and auto health checking nginx endpoints) + Nginx (webserver) + PHP 7.2 + MySQL (standard HA implementation with manual failover ATM, automation incoming) + Redis (Caching) + Elastic Search (search only, fails to MySQL if ES not available). Getting this running on AWS's flavor of everything really just sounds like asking to open a pretty big can of worms since the software (which I cannot modify freely) isn't built to do it.

I finished configuration management (hiccup the other day) and blacklab recently fixed the automated renewal of the ssl cert. A lot of the hiccups were due to being lazy and/or just not having the hardware we wanted in the rack (we had it in our garages)

As I mentioned soon we will be adding a high performance storage array to the rack and that'll give us a path forward on some cool stuff like image/media hosting for members.

 

See now... I keep trying to agree with you that you've got the costs in-hand, because you do, and you obviously have good reasons to do things this way, and my original comment was mostly in jest.

And then you say things like "Getting this running on AWS's flavor of everything..." Well now... AWS' flavor of everything is just an EC2 running HAProxy + an EC2 autoscaling group of EC2's running nginx + PHP 7.2 + MySQL RDS + Redis ElastiCache + AWS Elasticsearch. I mean it's the exact same thing you're running, except on AWS instead of on bare iron.

 

Link to comment
Share on other sites
Captainant

Captainant 27703

Posted
Posted
1 hour ago, immamac said:

Just so everyone knows it's running on HAProxy (LB now HA itself and auto health checking nginx endpoints) + Nginx (webserver) + PHP 7.2 + MySQL (standard HA implementation with manual failover ATM, automation incoming) + Redis (Caching) + Elastic Search (search only, fails to MySQL if ES not available). Getting this running on AWS's flavor of everything really just sounds like asking to open a pretty big can of worms since the software (which I cannot modify freely) isn't built to do it.

 

Let me preface everything I'm about to say with: I don't think you should change the site's architecture lol. But all this cloud talk has me wanting to flex nerd muscles and I think it's interesting to spitball.

So, if you were to migrate this workload to a cloud provider (I'll talk in AWS-speak for this), I would STRONGLY recommend:

  • ditch HAProxy in favor of ALB/ELB since it's a managed fucking loadbalancer and it just works. Also has tight integrations with autoscaling groups, API Gateway, etc.
  • ditch nginx webservers (which are really just handling HTTP and routing to your application) in favor of API gateway + lambda
  • migrate your application's code to a serverless architecture to leverage lambda + API gateway. I honestly don't know the specifics of Invision, but I can't imagine the code itself is storing any stateful info since it's backed by a DB
  • Fuck. Hosting. Your. Own. DB. Throw that shit into RDS, or even better, Aurora. Virtually infinite scalability, pay for exactly as much as you use, and never even have to worry about HA, backups, failovers, etc.
  • Use elasticache to cache, because managed beats running the infra yourself
  • Use a managed elastic search cluster, or you can completely ditch it if aurora has acceptable performance standalone (because it's significantly more durable in a managed service than self-hosted)

All that to say - there's definitely improvements and optimizations that could be realized without fundamentally changing how the site works, but there's also value in owning the physical boxes your shit is running on, and if there's a problem you can just go take care of it yourself.

On the other hand, we also have an aggy unplugging shit on accident 😛

Either way, thanks for running the site

Link to comment
Share on other sites
blacklab

blacklab 11941

Posted
Posted
6 hours ago, Captainant said:

ditch HAProxy in favor of ALB/ELB since it's a managed fucking loadbalancer and it just works. Also has tight integrations with autoscaling groups, API Gateway, etc.

I'm going to disagree with you here

With haproxy I can redirect http to https and host the certs inside it. I can also redirect surly to www.surly there as well, eliminating the need to do it on each box. I can also send different folders to different boxes.  

The answer above is based on what I'm allowed to use at work, I'm sure there's some super whamadine thing that I don't have access to that does all that.

aws is great if I'm not paying the bill, but I'm always the one getting the phone call from my boss on why our aws bill is so high and then have to track down what the idiot developers in our sister groups are doing. Like setting up a RDS cluster for a program that runs once a month for about 4 hours. There was no reason not to just have mysql running on the server that ran the script. So $2500 a month for 4 hours of usage and 696 hours of idling. 

I did interview at aws for a technical account manager role, and didn't get it, mainly because I got in an argument with one of the managers that iron is faster than virtual machines. Back at drillmap we had a big query that we managed to get from 20 seconds to .4 seconds through a bunch of different things, but mainly immamac getting mysql cluster going.  We did the same stuff at aws but it was always slower. I probably should not have told that story, but when he argued that it's impossible for iron to be faster than aws I knew i wasn't going to be a fit there.

I've been working on a project for work that involves Tb's of data. I have the biggest instance that aws offered a few months ago going, a 72 cpu monster with 192Gb of ram and dedicated ssd's. I processed a 100Gb file and then ran the same file on a machine that is now in the rack, a supermicro with dual  e5-2640 cpu's and 64Gb of ram and a cheapo ssd. The supermicro finished off the file in less than half the time the aws box did. 

I have spent dozens of hours screwing around on aws, figuring out if it would work for this site, but it always goes back to one thing, no matter what we do it will cost 3-7 times as much as we spend now, and that will take the site into a negative cash flow. 

 

Link to comment
Share on other sites
Captainant

Captainant 27703

Posted
Posted (edited)

I doubt that AWS will ever offer an instance type that will beat what you can host yourself in a rack. The whole point of it is to shape your workload to be able to scale horizontally. That O&G data file that's TB's in size is probably some binary abortion of a file that you have to load entirely before you can use it, and that doesn't scale well. If you tried to lift and shift that workload, of course it's not as good as your data center that's built custom for your workload. 

But, work out a compatibility layer between that dusty old monolith and a modern, scalable architecture, and you'll start to see some benefits. Going to the cloud isn't some silver bullet, it takes a bunch of work to make it worth it. 

I've got no doubt that running the site as-is is more expensive on AWS or any cloud provider. The whole idea is that your modify your workload to be less bespoke and more generic to take advantage of whatever the best tool is, rather than whatever fits your exact problem the best. 

But genuinely, I'm not saying you should run your site one way or another lol. You're the one doing the work and paying the bills, I just shitpost here. 

Edited by Captainant
Link to comment
Share on other sites
bamachine

bamachine 1231

Posted
Posted (edited)

Been trying to quote BL in the handover thread. Every post in that thread, hit quote and it takes me to top of the page. Then try to manually reply at bottom and it will not allow me to type anything in the reply box. I have noted that others do not have issues in that thread. Also the go to the top icon is missing from that thread.

 

Also, can we make those green banners at the top collapsible?

Edited by bamachine
  • Like 1
Link to comment
Share on other sites
Surly Bevo

Surly Bevo 25098

Posted
Posted
Been trying to quote BL in the handover thread. Every post in that thread, hit quote and it takes me to top of the page. Then try to manually reply at bottom and it will not allow me to type anything in the reply box. I have noted that others do not have issues in that thread. Also the go to the top icon is missing from that thread.
 
Also, can we make those green banners at the top collapsible?


Have had similar borking in the last day. Just refreshed page and it worked again.
Link to comment
Share on other sites
Pato del Muerto

Pato del Muerto 52543

Posted
Posted

On tapa to register this help ticket.

iPhone chrome user.

Can’t reply, can’t select my notifications in the top right.

The floating up arrow button that appears while scrolling to jump back to the top of the page doesn’t appear, but the jump to top is executed when I hit the insert other media button.

Can’t get a cursor in the reply field.

It’s your garden variety Bork-o-Rama.

Link to comment
Share on other sites

Join the conversation

You can post now and register later. If you have an account, sign in now to post with your account.

Guest
Reply to this topic...

×   Pasted as rich text.   Paste as plain text instead

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×
 Share
Go to topic listing


Football ... Basketball ... Baseball ... Other Sports ... Futbol ... 🤫995🤫 ... Gambling ... Movies & TV ... Music ... Hobbies ... Lulz ... Food & Travel ... Daily Texan ... Business and Markets ... Cloak Room ... Help ... For Sale ... Board Discussion ... Subscribe!... Donate!... Advertise... COOKIE MONSTER!

×
×
  • Create New...