Neil Turner's Blog

Blogging about technology and randomness since 2002


  1. How was it being blocked?

  2. I blocked it using two methods. 1 was this line in robots.txt:
    User-agent: ia_archiver
    Disallow: /
    And then in the .htaccess file:
    RewriteCond %{HTTP_USER_AGENT} ^ia_archiver
    RewriteRule .* – [F]
    This is, admittedly, a simplified version of what I actually use for blocking unwanted bots but you get the idea.

  3. how did you know you were blocking it?

  4. The ODP has recently updated its robots.txt file so that only a handful of bots are permitted. I checked my own as a matter of interest and noticed that Alexa’s bot was blocked.