Back up 9:54pm

February 25, 2008

WOWZA.  Ok, we’re back.


Something went wrong…

February 25, 2008

The site isn’t coming back up… the mysql server banned the apache server… fixing….


2/24 9:00pm Backing up database

February 25, 2008

If you see “Too many connections.” this is because the database got locked up during a backup.  This backup should be completed in about 5 minutes.  Sorry for the delays! 😦


DNS Fixed. Let the music play…

February 18, 2008

We got the DNS fixed and now everything is behaving nicely again.  All of the known problems from the previous post were fixed.  Rock on.


Fixing things, expect weird behavior (2/17/08)

February 17, 2008

There is an issue with our DNS servers (which help our servers know where other computers are), so a few weird things are messed up throughout the site.

Known issues include:

  • LyricWiki Challenge Facebook Application is completely unresponsive
  • iTunes Top Songs, Last.fm feeds and other RSS feeds on the site are not being pulled
  • Anything where the server accesses its own SOAP API (the client.php page or the “fixed” link on SOAPfailures) does not work

Teknomunk is l33t with this Linux stuff and is helping me work through it. Thanks for your patience, sorry about this 😦


Why our “stack” rules.

February 7, 2008

A “stack” is what tech people call their set of technologies that work together to run their application. It occurred to me recently how much LyricWiki‘s stack rules. We have only four servers and the web-server alone (which has an entire caching server in front of it, so this is just things that get past that) has been very comfortably handling more than 1.5 million pages per day. All of this and since I figured out this setup there haven’t been any slow-times even during the heaviest traffic. Did I mention this was only four commodity(cheap) servers!? How does it pull this off? Our stack is hardcore like Atreyu. Check it out:

The datacenter

The datacenter is a powerful beast at a great price. LyricWiki is hosted by G3 Technologies which has redundant everything and is on this cool Internet Exchange which puts our servers one-hop from an insane number of local Pittsburgh area people and campuses. We’re even one-hop away from Penn State (which is relatively far on a map). The fact that the place is so affordable is what has made it possible for LyricWiki to be supported only by non-invasive advertising.

The servers

LyricWiki is running on purely legal software for a total cost of $0. Win! Our operating system is CentOS which is basically the freely-compiled version of the same source-code as RedHat (G3 were the guys to tip me off to this; I naively hadn’t heard of CentOS 2 years ago). On top of the OS we have 3 different types of servers. A Squid caching server, an Apache web-server, and two mySQL servers (one is a replica/slave).

The application

The pages are written in PHP which is really efficient and is made much faster by APC op-code caching (this stores compiled versions of the code automatically since PHP is an interpreted language). The wiki itself is running the MediaWiki code (the same code that powers Wikipedia). I initially thought that MediaWiki was slow and bloated, but as time went on, I found out that in fact it is just optimized for very large wikis with a ton of traffic – this is because it was developed in conjunction with Wikipedia’s growth. One of my favorite features that was added into MediaWiki for scaling is the ability to instantly plug in the powerful in-memory object-caching system: memcached.

Other free stuff

On top of all of that, we’ve made heavy use of other technologies. ÜberBot, who added the first 200,000 or so songs to LyricWiki (to give it critical mass) was written in Perl which is fantastic at this sort of thing. We’ve also made search plugins for FireFox, Netscape, Safari, IE, etc. , a Facebook Application, and even leveraged the SOAP standard to make an API that has been used dozens if not hundreds of other places.

Third-party related tools

In addition to all of that stuff, we use Google Analytics and AWStats to track our stats.  The logo was made in the somewhat cumbersome but-hey-it’s-$400-cheaper-than-photoshop GIMP, and the code that I wrote for the site was written in Notepad++.  I’m sure I could go on and on listing other awesome (and almost always free) stuff that we use (IRC and even WordPress that I’m writing this on), but this post is getting long!

We use tons of amazing software and don’t have to pay for any of it! Brilliant! That’s kind of an interesting thing to notice being a programmer myself, but in the end that’s just something that the industry has to adapt to (much like the music industry with downloadable music).

I think I’ll leave you at that now. There was a TON of technology in there and I think I could write pages and pages about any one of the things I mentioned or linked to (except Atreyu, ironically), but that is for another day. If any of you are curious about any of the topics above, please comment on this post and it’s quite likely that I will expand up the topic later!

See how much our stack rocks? 🙂