Robots FAQ

This is a list with frequently asked questions about web robots. Select the question to go to the answer page, or select on the eye icon after the question to show the answer in this page.

1. About WWW robots
   • What is a WWW robot?
   • What is an agent?
   • What is a search engine?
   • What kinds of robots are there?
   • So what are Robots, Spiders, Web Crawlers, Worms, Ants?
   • Aren’t robots bad for the web?
   • Are there any robot books?
   • Where do I find out more about robots?
   • Any cartoons?
2. Indexing Robots
   • How does a robot decide where to visit?
   • How does an indexing robot decide what to index?
   • How do I register my page with a robot?
   • How do I get the best listing in search engines?
   • Can I use /robots.txt or meta tags to remove offensive content on some other site from a search engine?
3. For Server Administrators
   • How do I know if I’ve been visited by a robot?
   • I’ve been visited by a robot! Now what?
   • A robot is traversing my whole site too fast!
   • How do I keep a robot off my server?
4. Robots exclusion standard
   • Why do I find entries for /robots.txt in my log files?
   • How do I prevent robots scanning my site?
   • Where do I find out how /robots.txt files work?
   • What program should I use to create /robots.txt?
   • How do I use /robots.txt on a virtual host?
   • How do I use /robots.txt on a shared host?
   • What about further development of /robots.txt?
   • What if I cannot make a /robots.txt?
   • Can I block just bad robots?
   • Why did this robot ignore my /robots.txt?
   • Can a /robots.txt be used in a court of law?
   • Surely listing sensitive files is asking for trouble?
5. About META tags
   • What is the rel=”nofollow” link attribute?
6. Availability
   • Where can I use a robot?
   • Where can I get a robot?
   • Where can I get the source code for a robot?
   • I’m writing a robot, what do I need to be careful of?
   • I’ve written a robot, how do I list it?

Tags: FAQ, Robots