Error 502: Bad Gateway

Language: JP EN DE FR
users online
Forum » FFXIAH.com » News » Error 502: Bad Gateway
Error 502: Bad Gateway
 Fenrir.Scragg
Administrator
Offline
Server: Fenrir
Game: FFXI
user: Scragg
Posts: 2579
By Fenrir.Scragg 2013-01-03 12:46:00
Link | Quote | Reply
 
Error 502 Bad Gateway

This error occurs during overloaded situations, alot of websites have a custom "oops" page or something similar. Our webserver(Nginx) passes web pages requests to workers (in our case PHP) and if they are busy if will result in a 502.

Our old webserver(lighttpd) would handle this situation differently. It would make it appear the browser was trying to load the page and eventually timeout, as if the site wasn't even up.

I haven't worked on the site during the break but I'm aware of the 502 and addressing it, even before the break. I have audited the access logs for the site and found a few things. Firstly I am now blocking a few IPs found hammering the site. Google bots hit the site at about 4 reqs/second which the site can handle but the crawl rate is higher than what Google documented so I sent them a message. We get a ton of reqs from Guildwork plugin users. I isolated GW requests to a separate worker pool in hopes it doesn't interfere with the sites main worker pool. Each time an item tooltip is loaded, it makes a request. I configured Nginx to cache this so that it doesn't hit PHP at all (that was a big one). The worker pools had a backlog setting default of 128 requests, I upped it to 5000. The shout page is very popular, it appears some 3rd party program is accessing shout data. It should be blocked now. If someone wants shout data for 3rd party use, ask Guildwork.com I'm considering making that page for authenticated users only. I also started using the DB slave for achievement point calculations.

According to the logs through December, 502 would occur in 5-20min intervals. As of my changes it hasn't occurred, which is about 24 hours as of this post. Traffic has increased to the website in Dec due to a number of factors. The catch-22 in making the site faster is request volume can increase and you can find yourself back with the 502. Increasing volume is a good problem to have though. The site has plenty of hardware power for now so I'm focusing on optimizations. I'm not going to lie, the site can use plenty of optimizing.

This post is in response to a few people concerned with it and some thinking I'm not doing anything about it. This site isn't a wordpress, vbulletin or other cms that I can just throw a prepackaged caching solution or plugin in front of it. I aim to make Guildwork data, AH transactions and stock available as soon as it comes in. We are doing a lot of data collection and processing on the backend, over half of database queries are inserts! Considering the amount of reads we get, it's significant. When a 502 occurs, I will have it contact me and I will investigate.
[+]
 Bismarck.Magnuss
Offline
Server: Bismarck
Game: FFXI
Posts: 28615
By Bismarck.Magnuss 2013-01-03 12:49:03
Link | Quote | Reply
 
I never understood why people are so adamant about it. I found it to be a slight annoyance at its worst. But I just want to tell you good luck, Scragg. We're all counting on you.
[+]
 Odin.Godofgods
Offline
Server: Odin
Game: FFXI
Posts: 4013
By Odin.Godofgods 2013-01-03 12:50:44
Link | Quote | Reply
 
Bismarck.Magnuss said: »
I never understood why people are so adamant about it. I found it to be a slight annoyance at its worst. But I just want to tell you good luck, Scragg. We're all counting on you.

basicly.. yea'
Offline
By Aeyela 2013-01-03 12:51:24
Link | Quote | Reply
 
Thank you for posting about this, and I will be sure to regularly bump in an attempt to stop dozens of pointless threads from continuing to pop up.
 Asura.Bluespoons
Offline
Server: Asura
Game: FFXI
Posts: 5710
By Asura.Bluespoons 2013-01-03 12:52:25
Link | Quote | Reply
 
Thank god, our gate keeper showed up!
[+]
Offline
Posts: 42689
By Jetackuu 2013-01-03 12:58:38
Link | Quote | Reply
 
In my responses to other people I hope it didn't insinuate that I thought you weren't fixing it. I know from some talks we've had that you'll get to things when you can.

I'm going to start diving back into some web dev stuff, so if you need assistance let me know and I'll try.
 Bismarck.Kelhor
Administrator
Offline
Server: Bismarck
Game: FFXI
user: Rooks
Posts: 509
By Bismarck.Kelhor 2013-01-03 13:01:34
Link | Quote | Reply
 
Fenrir.Scragg said: »
over half of database queries are inserts

This is the craziest thing in that post, by far.
 Ragnarok.Ashman
Offline
Server: Ragnarok
Game: FFXI
user: Ashman
Posts: 4251
By Ragnarok.Ashman 2013-01-03 14:36:30
Link | Quote | Reply
 
I have an idea. We round off to the nearest cent. With all that incoming data, no one will be the wiser!
[+]
 Odin.Eikechi
Offline
Server: Odin
Game: FFXI
user: Eike
Posts: 9265
By Odin.Eikechi 2013-01-03 14:40:02
Link | Quote | Reply
 
Oh snap a topic started by one of the mighty red bars! Stuff gettin serious ^^. Thanks for the insight Scragg, and gl man.
 Caitsith.Zahrah
Offline
Server: Caitsith
Game: FFXI
user: zahrah
By Caitsith.Zahrah 2013-01-03 14:45:32
Link | Quote | Reply
 
Ragnarok.Ashman said: »
I have an idea. We round off to the nearest cent. With all that incoming data, no one will be the wiser!

Can we take the remaining fraction of each cent and hide it away? Maybe make a little interest off it too? Hmmm?

/devious
[+]
Offline
Posts: 346
By Sidiov 2013-01-03 15:37:44
Link | Quote | Reply
 
Caitsith.Zahrah said: »
Ragnarok.Ashman said: »
I have an idea. We round off to the nearest cent. With all that incoming data, no one will be the wiser!

Can we take the remaining fraction of each cent and hide it away? Maybe make a little interest off it too? Hmmm?

/devious
Yea, They did it in Superman 3.


My db insert for the day...
 Bismarck.Magnuss
Offline
Server: Bismarck
Game: FFXI
Posts: 28615
By Bismarck.Magnuss 2013-01-03 22:21:06
Link | Quote | Reply
 
I just want to tell you good luck, Scragg. We're all counting on you.
Log in to post.