Something is wrong…

Something is wrong on the server. It’s not slow in the normal sense caused by traffic. It’s completely bogged down: I could barely log in over SSH, it took a full minute just to run ‘ps’ and logging into mysql is taking a long… long time (still not done). I’ll update more once I figure out the problem.

…and I was all happy after the upgrades yesterday :-/

UPDATE1: I tried to restart the mysql server. I issued the command then waited… but it was taking so long that SSH disconnected me. I can’t reconnect, I’ve been trying on and off for about an hour. Bad. During the brief time that I was on I ran a SHOW PROCESSLIST command and noticed that part of the problem is coming from the fact that Plugins using the SOAP don’t stop making requests when the server is slow (humans on the site realize that it is slow or broken and stop browsing). So there were 269 queries running at once and the vast majority of them were SOAP requests. If the plugin you use has an option to turn it off temporarily, I would appreciate it if you do that! :]

UPDATE2:I called our webhost and they’re going to manually reboot the box and call right back, then I should be able to get everything working.

UPDATE3:The site is back up.  I’m pouring through logs and such trying to figure out what went wrong.  So far I’ve ruled out a DoS attack from any single source, so it appears that most likely traffic just got too high and my setting of max_connections was so high that things just went from bad to worse during a high-traffic time of day.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: