[subego]

Originally Posted by reader50 

I can diagnose some database errors, but these appear to be webserver load errors (or possibly LAN congestion). I've got only a basic load readout for the webservers, and zero access to dig into it. I'm hoping Mike is following this thread, or someone else with server access.

Incidentally Glen, the webserver renders pages anyway for admins, instead of throwing the error. A primary reason why we aren't seeing it. I used a test account the other day, and had the error pop up. It also ate the post I'd written, gratis.

Just make everybody an admin. What could go wrong?

[subego]

9:52-56a Central.

[subego]

10:56-8a Central.

[subego]

12:52p Central.

[subego]

12:57p Central.

[Thorzdad]

5:56 pm ET
And then 5:57 - 5:59

[subego]

7:55-6a Central.

Server's drinking early.

[ghporter]

Oh, we're special enough that we don't find out that stuff is going wonky? That's "special" alright. I almost never tinker with test users anymore, but maybe I need to start again, just to see what we're being "shielded" from...

I do think that there's some Internet traffic jamming going on at times. Some of the reported error times seem to coincide with what I would expect are prime congestion times; around 0700-0900 and 1530-1730 local to the user. Still, if it were just a traffic issue, wouldn't there be some sort of consistency in the frequency of errors?

[The Final Dakar]

Originally Posted by ghporter 

Still, if it were just a traffic issue, wouldn't there be some sort of consistency in the frequency of errors?

That's what's so odd. The first round was incredibly consistent. After they fixed it, it was consistent at a different time. But now its all over the place.

[subego]

If it's congestion, that's going to be at the server end.

Where is the server located?

I need the exact address. I can't get missile lock without it.

[Thorzdad]

11:56 am ET.

Originally Posted by subego 

Just make everybody an admin. What could go wrong?

[subego]

1:56p Central.

[besson3c]

Originally Posted by ghporter 

Oh, we're special enough that we don't find out that stuff is going wonky? That's "special" alright. I almost never tinker with test users anymore, but maybe I need to start again, just to see what we're being "shielded" from...

I do think that there's some Internet traffic jamming going on at times. Some of the reported error times seem to coincide with what I would expect are prime congestion times; around 0700-0900 and 1530-1730 local to the user. Still, if it were just a traffic issue, wouldn't there be some sort of consistency in the frequency of errors?

I believe the error message is triggered whenever there are 500 headers being returned by the web server. Do you have access to the logs? Let me know if you guys would like another set of eyes, although if memory serves me you guys don't have any real server access?

If you do, things to check on (in addition to your logs):

- overall load
- iostat/iowait
- PHP FPM/web server settings (there are several of these that relate to total processes)
- corrupt tables (mysqlcheck)
- operating system variables/sysctl that define the max number of open files

[ShortcutToMoncton]

6:02AM EST........can't imagine that is prime time. Main page says 140 active users and I'm the only registered one....

[Thorzdad]

Originally Posted by ShortcutToMoncton 

6:02AM EST........can't imagine that is prime time. Main page says 140 active users and I'm the only registered one....

That would be you and 139 bots of various types. That's a fairly normal number.

[subego]

7:01 and 7:08a Central.

[Thorzdad]

11:05 - 11:09 am ET.

[ghporter]

Originally Posted by besson3c 

I believe the error message is triggered whenever there are 500 headers being returned by the web server. Do you have access to the logs? Let me know if you guys would like another set of eyes, although if memory serves me you guys don't have any real server access?

If you do, things to check on (in addition to your logs):

- overall load
- iostat/iowait
- PHP FPM/web server settings (there are several of these that relate to total processes)
- corrupt tables (mysqlcheck)
- operating system variables/sysctl that define the max number of open files

I do not have access to server logs, and I don't think either reader50 or DemonHood have either. I agree that your suggestions for where to look for what's happening are good, and appropriate (though my server management knowledge is kinda limited to "ask the server guys to see what's happening"). But we have to ask "higher powers" to look into this sort of thing because we don't have the right access to the right stuff.

[besson3c]

Originally Posted by ghporter 

I do not have access to server logs, and I don't think either reader50 or DemonHood have either. I agree that your suggestions for where to look for what's happening are good, and appropriate (though my server management knowledge is kinda limited to "ask the server guys to see what's happening"). But we have to ask "higher powers" to look into this sort of thing because we don't have the right access to the right stuff.

I figured as much.

I also suspect that all of the posts here providing outage times will not ultimately be useful, because this sounds more like a service configuration issue than the result of actual organic traffic spikes. It's definitely cool that there are many people here that want to be helpful though

[arkayn]

0852 PDT this morning

[Thorzdad]

10:00 - 10:02pm ET tonight

[The Final Dakar]

This is still happening regularly for me

Originally Posted by reader50 

Do you have the ShowIP plugin installed (or equivalent) which shows the server IP you're talking to?

What would be the equivalent for Safari, and how would I use it to present a report?

Server too busy at 8:41am edt.

[arkayn]

And once again this morning at 0942 to 0944 PDT.

[Thorzdad]

Just got a Server Busy error coming here this morning (8:40am) Got one yesterday afternoon, too.
And, just got one trying to post this message (8:43)

[The Final Dakar]

It happening around 40 mark every hour.

[The Final Dakar]

So, same shit as over a year ago. Is this not correctable, requires money to correct, or somehow different in a subtle way?

[The Final Dakar]

Outlier at six minutes past the hour

[Mike Wuerthele]

(always watching and logging, no idea, passing to web team)

[The Final Dakar]

Repeat :06 performances. Haven't noticed if its in addition to 40 mark ones.

[Thorzdad]

I hit one around 9:50am this morning.

[Cap'n Tightpants]

10:50am today

[P]

There have been a few around the :50 mark recently.

[subego]

Interesting. I've been getting top of the hour for a few days.

[Thorzdad]

6:53pm EST tonight.

11:45am edt

[Thorzdad]

And 12:42-12:46

[Thorzdad]

8:43-8:51 am eastern, today

[Mike Wuerthele]

I'll get TPTB to do a reboot of the iron.

3:46pm edt, today

[Cap'n Tightpants]

4:15pm

and if you want to keep traffic up, don't close the threads that produce the most of it. Just sayin'.

[subego]

And let's just say, hypothetically, I was one of those people who brought news of the activity in those threads to the attention of the moderators.

I'd probably be pretty god damn discouraged by the result from doing so again in the future.

[ghporter]

Staying on topic (er hem...), I haven't had a server busy error in a long time. I usually visit around 0600 Central time, and then later around 1800 Central time. I wonder if there's a correlation between those specific times and less traffic, or maybe that these times are just naturally less busy because I get up (and get home) at an earlier hour than other people in this part of the US...

[subego]

Recently, mine have been all 45 past the hour, +/-3 minutes.

I've stopped reporting them because I'm almost positive it has nothing to do with an actual server load.

[Cap'n Tightpants]

3:00pm EST

[subego]

6:45a Central.

[subego]

8:51a Central.

[Thorzdad]

I've not run into a busy signal this morning, but, around 8:30 am eastern, this place was running criminally slow across the board.

[subego]

11:50a Central.