PDA

View Full Version : robots.txt seems to be ignored :?


galacticvoyager
11-12-2005, 11:55 AM
While PHPDIG is spidering my site it reports the follow:

48:http://www.galacticvoyager.com/bb/phpBB2/index.php
(time : 00:00:37)
+ + + + + + + + + + + + + + + + + + + + +

I thought this meant it was indexing the given URL however my robots.txt file look like this:

User-agent: *
disallow: /bb/
disallow: /pat/
disallow: /mod/

Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle?

Thanks for your help.

Charter
11-12-2005, 12:52 PM
In the PhpDig v.1.8.8 phpdigReadRobotsTxt function in robot_functions.php find:

if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {

And replace with:

if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {


In the PhpDig v.1.8.9 RC1 phpdigReadRobotsTxt function in robot_functions.php find:

if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {

And replace with:

if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {