robots and spiders
15th Feb 2005
Is it just me or has there been an enormous increase in the amount of spidering carried out by robots like msnbot/1.0, Googlebot/2.1 and Yahoo! Slurp?
Possibly related?
Based on the tags from this post.
Robots Exclusion
Web Server Administrator's Guide to the Robots Exclusion Protocol. This page gives you a run-through of what to include in your robots.txt file. H [...]
- up ↑
- web design
- 5th Feb 2004
agents and spiders and bots, oh my!
Just been checking through the referrers from Doepud and found a few interesting links.Here's a cool looking search engine: Seekport. I like the previ [...]
- up ↑
- web design
- 13th May 2005
