September 12, 2008
This week, the site has been extremely slow and even gone down and up a couple of times. I searched for a problem for a while but it appears that we’ve just really hit the wall on how much traffic we can support with our current servers. That’s fairly good timing since we’d been planning to move to more servers for a little while, so I’d already begun to look into it.
Today I ordered another server with the same specs as the current Apache server. This will bring us up to 5 total servers running LyricWiki. For the curious (and tech-savvy): that’s one squid caching server in front of two Apache web servers which talk to one mysql master server and one read-only replica mysql server.
To get the server to be as beefy as we need, I had to ask the hosting company to order extra RAM for it. So we’re just waiting for that to be delivered (hopefully around this weekend or very soon after) and we’ll be ready to start working to get the new server pulled into our setup.
In addition to just having more man-power machine-power to handle our traffic, this will give two additional benefits immediately. The first is that we can use the new server to test out the upgrade to the newest version of MediaWiki (the software that runs our site as well as Wikipedia). The second benefit is that now we’ll have two Apache servers – currently the most overworked part of the system – with one running the API and one running the site itself (lyricwiki.org). This will let us more quickly identify when something is wrong with one of those two systems and it will make sure that problems with either of them are unlikely to effect the other.
Exciting times… stay tuned!
July 27, 2008
Finally you can have your watchlist stream you content when the pages you are interested in have been updated!
Thanks to a new extension by Teknomunk, you can now enable the rss feed at http://lyricwiki.org/Special:WatchlistFeed.
Since this is a really great feature to have in any wiki, Teknomunk plans to make this extension public so that all MediaWikis can use this feature (it is very commonly requested). Thanks Teknomunk! 🙂
March 30, 2008
I’m proud to announce a long-overdue feature: implied redirects.
Implied redirects make it so that the site can often understand what you’re looking for even if it is misspelled or we don’t have a redirect page for the specific song. For example, we have a redirect from the band name “Of A Revolution” to their preferred form “O.A.R.“. However, if someone comes to the site, we do not have a redirect from “Of A Revolution:Crazy Game Of Poker” to the correct page: “O.A.R.:Crazy Game Of Poker“. With the new implied-redirects extension, the site will automatically figure out what you meant and display the correct page. To see it in action, go to “Of A Revolution:Crazy Game Of Poker“.
Implied redirects have been active in the API for quite some time, but didn’t work on the site until tonight.
March 29, 2008
LyricWiki is now in Wikipedia’s inter-wiki map.
To decrypt that for most users: now if you want to create a link from a wikipedia page to a LyricWiki page, all you have to do is enter [[lyricwiki:Page Name]] on the page and it will make the link for you.
Here is an example of what you could put in the wikipedia page for the band Tool which would link to the lyrics on LyricWiki:
Pretty easy, huh? The “Tool lyrics” part is just the text that you want to have show up as the link. When you think a wikipedia article would benefit from linking to a LyricWiki page, feel free to throw one of those inter-wiki links in there.
March 13, 2008
We’re going to be upgrading LyricWiki to the newest version of MediaWiki. This is good for a number of reasons, not the least of which is that the newest versions always have the lowest number of known security vulnerabilities.
There are also some new features that our extensions can’t work without.
Teknomunk was kind enough to jump right in and test/re-write a bunch of the extensions (I think all of them actually) to make sure they work in the new version. He also made some upgrades which only work in the new version (which was what ultimately caused the decision to upgrade… so give him mad props :)).
Anyway… this is going to cause a lot of outages tonight. I’m going to shut down the API (the SOAP), back up the database, then do the upgrades, fix what’s broken, then turn the API back on. I’ll keep this blog updated as things progress.
November 16, 2007
Long story short: everything was backwards but it is fixed now.
For the curious, here are the technical details:
Although undocumented, the MediaWiki doesn’t just use your normal one-line database settings for your master server… if you have multiple servers set up, it uses the server at index 0 as the master. Our “slave” was at index 0 with 99% of the read load until yesterday. Then it was still at index 0 but with only 70% of the read load.
The effect this had was that the powerful server just kind of sat there with stale data serving 1% of reads until yesterday, when it started serving 30% of the reads, showing it’s ugly out-of-date data to the world.
I copied the data from the “slave” to the master and changed the configuration so that it really is the master again. Most of the data (except for a few minutes of changes right before the switch that were mostly by myself and another admin) should still be in tact.
Whatamess that was. Thanks to all of the admins and contributors on LyricWiki for pointing these problems out to me, and thanks to TimStarling of WikiMedia and domas of MySQL for their help figuring out what was wrong and helping me fix it. You guys are ninjas.
November 15, 2007
If you’ve been using the API over the past week, you probably noticed the painfully large percentage of results that were being returned as “Not found”.
The large increase in traffic recently was causing us to get “Too many connections” errors when the API was left alone, so had to turn on a throttling system which would randomly drop a certain percentage of the requests. Looking into our server logs, I found out that our actual web server (behind our Squid caching server which serves up 30% of our pages) has been getting over 1 million page requests per day! Wow… that explains the scaling problems.
I was overly busy for most of the week (a drawback of having LyricWiki not be my “day-job”), so I first got to really attack the problem tonight. It appears that everything is back up to speed, and the throttling is turned off. I’ll be keeping an eye on how the site is doing tomorrow during peak traffic time, but I think we should be okay.
I have some more fixes planned for the near-future which should make it so the API can continue to handle increasing traffic. I probably won’t post about them as they happen, but hopefully you’ll notice an increase in the speed that results are served up.