Caught back up

August 31, 2008

The slave database server is right on pace now.  This means that there should no longer be delays between when you make an edit and when it shows up.

Very cool.


Potential slowness and strange updates

August 30, 2008

A major template was modified today, this caused almost every page to need to be thrown away from the cache.  This has the side-effect of making our slave-database (which is a copy of the main database) get way behind where it is supposed to be since there is a bunch of other work for the server to do.  This means that edits you make today might take a little while before they show up.

Currently, the slave is falling further and further behind.  It’ll probably continue to get worse before it gets better, but hopefully we should be okay in a day or two.

Sorry for the inconvenience!

Back up…

August 16, 2008

Problem fixed.

Coincidentally, today I’m working on a long-term plan to fix everything I consider a weak-point of LyricWiki and uptime and manual monitoring is certainly one of those (the reason it tends to go down the most when I’m asleep is that I’m physically checking in on what’s happening on the servers while I’m awake so I catch most things before they turn bad).

Site down… back soon…

August 16, 2008

It appears to have been down since around 6:20.  Working on it…

Finally fast again.

August 10, 2008

For the first time all day, the site is moving at what appears to be full-speed.

Also, to answer the earlier-posed question about the Squid having to log a ton of extra space when it restarts, it turns out that is true – apparently something (either Apache restarting or somehow detecting that the Squid just came back) triggers it so that the MediaWiki install sends a <em>ton</em> of “PURGE” requests to the Squid server which basically tell it to forget about a page it may be caching because it’s probably out of date now.  Each request is another line in the log-file, so that’s about 800,000 extra lines in the span of a few minutes.

Squid ran out of space again this morning (fixed now)

August 10, 2008

Apparently the site was down while I slept, but I had emails & comments in my inbox about the outage as soon as I got up so I was able to jump on it right away.

The Squid had run out of memory again.  The access log files for one day were 17 gigs.  That seems awfully high – maybe we’re getting spidered too hard or the logs go through serious stress after a restart?

I’ll be finding a more permanent solution to the issue, but in the meantime the site is “up” but it’s going to be fairly slow while the cache refills… again.

Squid back up :)

August 9, 2008

It had run out of harddrive space.   Moved some log files to another disk and we’re back in business.