April 25, 2009
We delayed the previous datacenter move and adjusted it to a better time of day: 7am EST on a Sunday.
The outage should be between 30 minutes and an hour if everything goes smoothly and indefinitely longer if it doesn’t go smoothly!
Some additional info that hasn’t been on the blog: we were generously given two more servers by a local company that was upgrading (thanks!) and have ordered gobs of RAM for them. Once that is in, we will be upgrading LyricWiki‘s setup once again. We currently have 6 servers, this will bring us up to 8! That’s a significant increase and hopefully will make LyricWiki even faster.
February 26, 2009
The problems earlier today were apparently caused by an attempted distributed denial of service attack.
The datacenter has got all of this bad traffic blocked now, and the site appears to be back to normal (for several hours in a row now). The whole problem lasted about 1 to 2 hours but it’s tough to measure since the site was never completely down during that time.
February 26, 2009
The site is currently behaving very sporadically… there are some problems in the connection between our datacenter (where all of our servers hang out) and the internet.
Not sure whether this is related to the “emergency maintenance” Sprint was doing last night or not.
The team at the datacenter is working on it right now, and hopefully we’ll be back soon.
September 20, 2008
The new server exists, I’m setting it up and am hoping I can have it all done before I go to sleep.
There is a decent amount of stuff that needs to be done before it’s all working. If you’re technical, or just like watching lists as they are being completed… I’ll be tracking the upgrade here.
I’ll also be making occasional updates to the @lyricwiki twitter account.
The new server will be another Apache server and is named “Cochise“. That is the name of the last great Apache chief, and also the inspriation for the song Cochise by Audioslave.
There will be downtime for an unknown length of time tonight. I’ll try to keep it to a minimum, but the site is so slow it’s practically down anyway.
PS: Special thanks to our awesome webhost for getting the new box here & set up quickly!
September 19, 2008
And not a moment too soon. Traffic is down a significant percentage in the last few days. The site is so slow and bothersome that I even find myself hesitating to load more pages when I go to the site.
Anyway, while I was waiting, I made a list of all of the changes I think are necessary for the upgrade (although there is the potential that I could get surprised). I plan to update the list during the upgrade. There will be an outage during which the list will be unavailable, but that’s not a huge deal, hopefully. Some of the tasks don’t require the new server to be here for them to be started, so I’ve already begun.
Here is the list of upgrade tasks: http://lyricwiki.org/upgradeList.html
ps: If anyone wants to make a style-sheet to make the page less ugly, I’d be more than happy to put it up, but designing one is lower-priority for me than doing the things on the list.
September 12, 2008
This week, the site has been extremely slow and even gone down and up a couple of times. I searched for a problem for a while but it appears that we’ve just really hit the wall on how much traffic we can support with our current servers. That’s fairly good timing since we’d been planning to move to more servers for a little while, so I’d already begun to look into it.
Today I ordered another server with the same specs as the current Apache server. This will bring us up to 5 total servers running LyricWiki. For the curious (and tech-savvy): that’s one squid caching server in front of two Apache web servers which talk to one mysql master server and one read-only replica mysql server.
To get the server to be as beefy as we need, I had to ask the hosting company to order extra RAM for it. So we’re just waiting for that to be delivered (hopefully around this weekend or very soon after) and we’ll be ready to start working to get the new server pulled into our setup.
In addition to just having more man-power machine-power to handle our traffic, this will give two additional benefits immediately. The first is that we can use the new server to test out the upgrade to the newest version of MediaWiki (the software that runs our site as well as Wikipedia). The second benefit is that now we’ll have two Apache servers – currently the most overworked part of the system – with one running the API and one running the site itself (lyricwiki.org). This will let us more quickly identify when something is wrong with one of those two systems and it will make sure that problems with either of them are unlikely to effect the other.
Exciting times… stay tuned!
July 27, 2008
Finally you can have your watchlist stream you content when the pages you are interested in have been updated!
Thanks to a new extension by Teknomunk, you can now enable the rss feed at http://lyricwiki.org/Special:WatchlistFeed.
Since this is a really great feature to have in any wiki, Teknomunk plans to make this extension public so that all MediaWikis can use this feature (it is very commonly requested). Thanks Teknomunk! 🙂