![]() |
|
![]() |
#1 |
Green Mole
Join Date: Nov 2005
Posts: 3
|
robots.txt seems to be ignored :?
While PHPDIG is spidering my site it reports the follow:
48:http://www.galacticvoyager.com/bb/phpBB2/index.php (time : 00:00:37) + + + + + + + + + + + + + + + + + + + + + I thought this meant it was indexing the given URL however my robots.txt file look like this: User-agent: * disallow: /bb/ disallow: /pat/ disallow: /mod/ Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle? Thanks for your help. |
![]() |
![]() |
![]() |
#2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
In the PhpDig v.1.8.8 phpdigReadRobotsTxt function in robot_functions.php find:
Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) { Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) { Code:
if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) { Code:
if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
![]() |
![]() |
![]() |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
robots.txt and URL | djavet | How-to Forum | 4 | 01-11-2005 03:19 AM |
robots.txt comments | edkay | Mod Submissions | 2 | 03-12-2004 12:41 PM |
robots.txt versus robotsxx.txt | Charter | IPs, SEs, & UAs | 0 | 03-11-2004 06:00 PM |
robots.txt ignored | roy | Troubleshooting | 3 | 02-20-2004 08:02 PM |
robots.txt | renehaentjens | Troubleshooting | 3 | 12-05-2003 02:40 PM |