robots.txt seems to be ignored :?
While PHPDIG is spidering my site it reports the follow:
48:http://www.galacticvoyager.com/bb/phpBB2/index.php (time : 00:00:37) + + + + + + + + + + + + + + + + + + + + + I thought this meant it was indexing the given URL however my robots.txt file look like this: User-agent: * disallow: /bb/ disallow: /pat/ disallow: /mod/ Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle? Thanks for your help. |
In the PhpDig v.1.8.8 phpdigReadRobotsTxt function in robot_functions.php find:
Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) { Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) { Code:
if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) { Code:
if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) { |
All times are GMT -8. The time now is 06:20 PM. |
Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.