View Single Post
Old 02-20-2004, 01:56 PM   #1
roy
Green Mole
 
Join Date: Feb 2004
Posts: 3
robots.txt ignored

I have a site where my robots.txt acts as a honeypot to block bots that dont obey the standard.

This is what my robots.txt looks like:

User-agent: *
Disallow: /elguapo/index.php

I even added this to it now:

User-agent: PhpDig
Disallow: /elguapo

Still, phpdig, ends up ripping thru all the pages, ignores my robots.txt and gets banned. I see nothing in the excludes table.

Is it supposed to be writing to it? How do I get phpdig to obey the robots directives?
roy is offline   Reply With Quote