Neil Turner's Blog

Blogging about technology and randomness since 2002

Restless Robots

My stats are now showing over 3500 individual 404 errors over the past 12 days. I’ve checked as many links as possible, yet I couldn’t put my finger on what is causing all of these errors.
Then an idea come from a comment Richy left on my customised 404 page article. A lot of his 404 errors come from browsers looking for favicon.ico. Now while the public pages use a link statement to specify a PNG image as the icon (since PNG is a better format), the Movable Type scripts do not, and I imagine that every time I load a page, Mozilla goes looking for this file, and of course doesn’t find it. So I’ve added it.
Then I remembered seeing in the referral logs a few attempts to access robots.txt, which also doesn’t exist (although I did add one to the old site back in August). Bearing in mind that the Googlebot is the third most popular user agent, taking in 12% of hits, I imagine that it generates a 404 every time it reaches this site, so that’s a lot of 404s. Teoma and Inktomi’s bots also rank highly, taking a further 4% of hits with them.
Which reminds me, I’ll be resetting the referral logger at the end of the month. The database is nearly 1MB now, with only 4MB remaining on the server, and I imagine most of you don’t look much beyond the first 50 entries anyway.

Comments are closed.