robots and spiders
15th Feb 2005
Is it just me or has there been an enormous increase in the amount of spidering carried out by robots like msnbot/1.0, Googlebot/2.1 and Yahoo! Slurp?
Based on the tags from this post.
Web Server Administrator's Guide to the Robots Exclusion Protocol. This page gives you a run-through of what to include in your robots.txt file. H [...]
Type about:robots into the address bar of Firefox 3. Also, try about:mozilla. [...]
Robots in Disguise
This band sound and look pretty good and their site in rather nifty too! Check out Robots in Disguise. [...]
This could be good if they manage to get their subscription form working correctly... more decent music for free download at Modular Music, link vi [...]
agents and spiders and bots, oh my!
Just been checking through the referrers from Doepud and found a few interesting links.Here's a cool looking search engine: Seekport. I like the previ [...]
A bunch of music blogs I'm goin to check out quite soon:http://teachingtheindiekidstodanceagain.blogspot.com/http://www.fatplanet.co m.au/http://mus [...]