robots.txt ignored
I have a site where my robots.txt acts as a honeypot to block bots that dont obey the standard.
This is what my robots.txt looks like:
User-agent: *
Disallow: /elguapo/index.php
I even added this to it now:
User-agent: PhpDig
Disallow: /elguapo
Still, phpdig, ends up ripping thru all the pages, ignores my robots.txt and gets banned. I see nothing in the excludes table.
Is it supposed to be writing to it? How do I get phpdig to obey the robots directives?
|