Move completed

We’re back.  All of the servers are moved and everything is back to normal.

If the servers ever reboot now, they’ll automatically function (not at full speed though… still working on how to autostart memcached the right way).  It shouldn’t even matter what order they come up in thankfully.  Good coding by Squid, Apache, and mySQL programmers!     LAMP stack rules.

Anywho… have a nice night.

Move to new rack is underway…

Unknown amount of downtime… all four servers have to be moved.  I’ll be here waiting for them to go back online, then I’ll turn everything back on.

Whew… chilled out.

So the problem that was invisible to users last night, but mandated that we reboot the server this morning (which did not go smoothly) was all because of a typo on my part yesterday.  I was installing a new ad-server, and basically I switched some parameters and a command started doing some bad things.  Thankfully the dudes at G3 helped me out, and once we figured out what was wrong it became clear to me exactly what I had done wrong and how it went amok.

If you desperately needed lyrics between like 10 and 11 and you couldn’t get them… that was my fault!   I am sorry, and I’ll try not to be a n00b again.

I’m also going to be making a script so that after a reboot, the entire system re-configures itself automatically so that we don’t have significant downtimes (I think the machines can reboot in like 30 seconds).

Now that the stress is over… here is some chill music.

Server on the mend… site back up

The site is back up.  It’s probably really slow as all of the caching mechanisms start storing things again.  I actually have to turn some of them back on and do a little more grooming to get it going smoothly & fullspeed again.  I’ll give more info later.

Rebooting… unknown restart time.

Something didn’t work correctly on the reboot… I’m trying to figure it out now.

Correction: downtime rescheduled for 1/17/08

The server isn’t going down tonight… instead I’ll wait until tomorrow because our web host
is putting my servers into new racks (for free… shwing!).  There’s going to be some downtime for that (gotta move 4 servers), and probably a brief outage tomorrow morning/early-afternoon to do that little fix I was going to do tonight.

Ripping off the band-aid… getting all of the downtime out of the way tomorrow hopefully   (haven’t had any since Thanksgiving until this I don’t think).

After I’m in, I’ve also started working on a way to make downtime even shorter in the rare occasions that we will have to restart the server in the future (previous downtime was primarily due to the surges in traffic which we finally figured out how to handle).

Breif outage coming on 1/16

We have to fix a hiccup w/one of the servers.  The site will be down for a few minutes.

Thanks for your patience… I’ll get it back up as soon as I can!