This week, the site has been extremely slow and even gone down and up a couple of times. I searched for a problem for a while but it appears that we’ve just really hit the wall on how much traffic we can support with our current servers. That’s fairly good timing since we’d been planning to move to more servers for a little while, so I’d already begun to look into it.
Today I ordered another server with the same specs as the current Apache server. This will bring us up to 5 total servers running LyricWiki. For the curious (and tech-savvy): that’s one squid caching server in front of two Apache web servers which talk to one mysql master server and one read-only replica mysql server.
To get the server to be as beefy as we need, I had to ask the hosting company to order extra RAM for it. So we’re just waiting for that to be delivered (hopefully around this weekend or very soon after) and we’ll be ready to start working to get the new server pulled into our setup.
In addition to just having more man-power machine-power to handle our traffic, this will give two additional benefits immediately. The first is that we can use the new server to test out the upgrade to the newest version of MediaWiki (the software that runs our site as well as Wikipedia). The second benefit is that now we’ll have two Apache servers – currently the most overworked part of the system – with one running the API and one running the site itself (lyricwiki.org). This will let us more quickly identify when something is wrong with one of those two systems and it will make sure that problems with either of them are unlikely to effect the other.
Exciting times… stay tuned!