We’re back!

For the few ppl who read this blog… you may now return to your normal lives: http://lyricwiki.org

Should be back soon…

All of the servers are moved now, just trying to get things to work again.

Scheduled downtime Apr. 26th, 2009 at 7am EST

We delayed the previous datacenter move and adjusted it to a better time of day: 7am EST on a Sunday.

The outage should be between 30 minutes and an hour if everything goes smoothly and indefinitely longer if it doesn’t go smoothly!

Some additional info that hasn’t been on the blog: we were generously given two more servers by a local company that was upgrading (thanks!) and have ordered gobs of RAM for them. Once that is in, we will be upgrading LyricWiki’s setup once again. We currently have 6 servers, this will bring us up to 8! That’s a significant increase and hopefully will make LyricWiki even faster.

Scheduled downtime: March 21, 2009 at 3pm EST

Our webhost is moving to a new datacenter right down the street from the current datacenter.  We’ve scheduled with them to move the LyricWiki servers at 3pm tomorrow, March 21st.  It will take slightly under a half an hour to actually move the servers (during which time the site will be down completely), then we will try to get everything back up and running as soon as possible.

If there are any major problems, we will post about them here.  For more up-to-the-minute info, you can follow the @lyricwiki twitter account or join us on the #LyricWiki IRC channel on quakenet.

Back to normal after DDoS attack

The problems earlier today were apparently caused by an attempted distributed denial of service attack.

The datacenter has got all of this bad traffic blocked now, and the site appears to be back to normal (for several hours in a row now).  The whole problem lasted about 1 to 2 hours but it’s tough to measure since the site was never completely down during that time.

2/26: Site will be up and down

The site is currently behaving very sporadically… there are some problems in the connection between our datacenter (where all of our servers hang out) and the internet.

Not sure whether this is related to the “emergency maintenance” Sprint was doing last night or not.

The team at the datacenter is working on it right now, and hopefully we’ll be back soon.

Most listened-to songs of 2008

A post on LifeHacker mentioned that Last.fm has released their lists of the most listened-to songs of 2008.  Since their service “scrobbles” (logs) everything that their users listen to, they have a pretty massive data-set.  They are UK-based, so there is quite a bit of a UK bias in the results, but they are still interesting nonetheless.

There were two somewhat-annoying things about the lists: 1) each entry is on a different page, so you have to go to 30 pages to see all of the results 2) no links to lyrics!

So we’ve taken the liberty of compiling them for you into a nice concise list of links to the lyrics-pages:

http://lyricwiki.org/LyricWiki:Lists/2008/Last.fm

Enjoy!

New server looks good (plus a surprise!)

So far, LyricWiki has been running pretty well with the new server (our 5th server, named “Cochise“) set up with the rest of them.  I’ve moved the API completely to that server which takes a good deal of the stress off of the site itself.

The site has been pretty fast since the new server has been up.  However, there have been occasional slow-patches, and looking at the CPU usage on the main webserver for the site – it’s definitely still not “calm”.

Surprise 1

So here’s a fun surprise: today I ordered another server!

For the first time in a looong time, we can hopefully stay ahead of demand instead of suffering for a couple of weeks until it’s unbearable and we’re forced to upgrade.

Surprise 2

W00t! This is an uplifting blogpost… lots of surprises! Anyway: the second surprise is that starting in October, I’m going to be decreasing the time I spend at my day-job by 40% so I’ll only be in the office 3 days per week. That’ll give me two whole days more each week where I can work on Motive Force products like LyricWiki. This should help things become much more stable very quickly.

Well, that was an abnormally enjoyable post for this blog – which usually just announces outages! The site still isn’t totally upgraded (because the weekend ended before I could make all of the extensions work), so I have to run back to that. I took snapshots of a bunch of performance stats before and after, so I’ll post those sometime soon.

Thanks for your patience during those past few weeks. Hopefully we’re in a whole new era for the site!

Setting up new server right now

The new server exists, I’m setting it up and am hoping I can have it all done before I go to sleep.

There is a decent amount of stuff that needs to be done before it’s all working. If you’re technical, or just like watching lists as they are being completed… I’ll be tracking the upgrade here.

I’ll also be making occasional updates to the @lyricwiki twitter account.

The new server will be another Apache server and is named “Cochise“. That is the name of the last great Apache chief, and also the inspriation for the song Cochise by Audioslave.

There will be downtime for an unknown length of time tonight. I’ll try to keep it to a minimum, but the site is so slow it’s practically down anyway.

PS: Special thanks to our awesome webhost for getting the new box here & set up quickly!

New server coming tomorrow!

And not a moment too soon.  Traffic is down a significant percentage in the last few days.  The site is so slow and bothersome that I even find myself hesitating to load more pages when I go to the site.

Anyway, while I was waiting, I made a list of all of the changes I think are necessary for the upgrade (although there is the potential that I could get surprised).  I plan to update the list during the upgrade.  There will be an outage during which the list will be unavailable, but that’s not a huge deal, hopefully.  Some of the tasks don’t require the new server to be here for them to be started, so I’ve already begun.

Here is the list of upgrade tasks: http://lyricwiki.org/upgradeList.html

ps: If anyone wants to make a style-sheet to make the page less ugly, I’d be more than happy to put it up, but designing one is lower-priority for me than doing the things on the list.