Possibly related?
Based on the tags from this post.
Robots Exclusion
Web Server Administrator's Guide to the Robots Exclusion Protocol. This page gives you a run-through of what to include in your robots.txt file. H [...]
- up ↑
- web design
- 5th Feb 2004
Closing the gap - User Agent improvements
AlastairC talks about improvements that could be made with User Agents including, most interestingly, the idea of setting up 'Profiles'. The idea behi [...]
- up ↑
- accessibility
- 31st May 2007
Mood change for P2P?
Article on The Register about recent changes in the peer-to-peer business. [...]