PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   Bug Tracker (http://www.phpdig.net/forum/forumdisplay.php?f=27)
-   -   robots.txt seems to be ignored :? (http://www.phpdig.net/forum/showthread.php?t=2247)

galacticvoyager 11-12-2005 11:55 AM

robots.txt seems to be ignored :?
 
While PHPDIG is spidering my site it reports the follow:

48:http://www.galacticvoyager.com/bb/phpBB2/index.php
(time : 00:00:37)
+ + + + + + + + + + + + + + + + + + + + +

I thought this meant it was indexing the given URL however my robots.txt file look like this:

User-agent: *
disallow: /bb/
disallow: /pat/
disallow: /mod/

Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle?

Thanks for your help.

Charter 11-12-2005 12:52 PM

In the PhpDig v.1.8.8 phpdigReadRobotsTxt function in robot_functions.php find:
Code:

        if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {
And replace with:
Code:

        if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {
In the PhpDig v.1.8.9 RC1 phpdigReadRobotsTxt function in robot_functions.php find:
Code:

        if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {
And replace with:
Code:

        if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {


All times are GMT -8. The time now is 10:13 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.