Back to normal after DDoS attack

The problems earlier today were apparently caused by an attempted distributed denial of service attack.

The datacenter has got all of this bad traffic blocked now, and the site appears to be back to normal (for several hours in a row now).  The whole problem lasted about 1 to 2 hours but it’s tough to measure since the site was never completely down during that time.

2/26: Site will be up and down

The site is currently behaving very sporadically… there are some problems in the connection between our datacenter (where all of our servers hang out) and the internet.

Not sure whether this is related to the “emergency maintenance” Sprint was doing last night or not.

The team at the datacenter is working on it right now, and hopefully we’ll be back soon.

Squid back up :)

It had run out of harddrive space.   Moved some log files to another disk and we’re back in business.

Taking another crack at it… site will be down.

Once more into the breach!

I’m going to hurl myself at the database-replication problem again.  The current plan I have unfortunately involves some downtime for the site.  But it’s already after 1am so I’m hoping the impact on users is relatively low.

I’ll keep you updated.

3/12/08 9:30pm EST: Upgrading LyricWiki – expect outages.

We’re going to be upgrading LyricWiki to the newest version of MediaWiki.  This is good for a number of reasons, not the least of which is that the newest versions always have the lowest number of known security vulnerabilities.

There are also some new features that our extensions can’t work without.

Teknomunk was kind enough to jump right in and test/re-write a bunch of the extensions (I think all of them actually) to make sure they work in the new version.  He also made some upgrades which only work in the new version (which was what ultimately caused the decision to upgrade… so give him mad props ).

Anyway… this is going to cause a lot of outages tonight.  I’m going to shut down the API (the SOAP), back up the database, then do the upgrades, fix what’s broken, then turn the API back on.   I’ll keep this blog updated as things progress.

Correction: downtime rescheduled for 1/17/08

The server isn’t going down tonight… instead I’ll wait until tomorrow because our web host
is putting my servers into new racks (for free… shwing!).  There’s going to be some downtime for that (gotta move 4 servers), and probably a brief outage tomorrow morning/early-afternoon to do that little fix I was going to do tonight.

Ripping off the band-aid… getting all of the downtime out of the way tomorrow hopefully   (haven’t had any since Thanksgiving until this I don’t think).

After I’m in, I’ve also started working on a way to make downtime even shorter in the rare occasions that we will have to restart the server in the future (previous downtime was primarily due to the surges in traffic which we finally figured out how to handle).