PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Bug Tracker

Reply
 
Thread Tools
Old 11-12-2005, 11:55 AM   #1
galacticvoyager
Green Mole
 
Join Date: Nov 2005
Posts: 3
robots.txt seems to be ignored :?

While PHPDIG is spidering my site it reports the follow:

48:http://www.galacticvoyager.com/bb/phpBB2/index.php
(time : 00:00:37)
+ + + + + + + + + + + + + + + + + + + + +

I thought this meant it was indexing the given URL however my robots.txt file look like this:

User-agent: *
disallow: /bb/
disallow: /pat/
disallow: /mod/

Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle?

Thanks for your help.
galacticvoyager is offline   Reply With Quote
Old 11-12-2005, 12:52 PM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
In the PhpDig v.1.8.8 phpdigReadRobotsTxt function in robot_functions.php find:
Code:
        if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {
And replace with:
Code:
        if (eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {
In the PhpDig v.1.8.9 RC1 phpdigReadRobotsTxt function in robot_functions.php find:
Code:
        if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?($allowed_link_chars))',$line,$regs)) {
And replace with:
Code:
        if (mb_eregi('[[:blank:]]*disallow:[[:blank:]]*([/]?('.$allowed_link_chars.'))',$line,$regs)) {
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
robots.txt and URL djavet How-to Forum 4 01-11-2005 03:19 AM
robots.txt comments edkay Mod Submissions 2 03-12-2004 12:41 PM
robots.txt versus robotsxx.txt Charter IPs, SEs, & UAs 0 03-11-2004 06:00 PM
robots.txt ignored roy Troubleshooting 3 02-20-2004 08:02 PM
robots.txt renehaentjens Troubleshooting 3 12-05-2003 02:40 PM


All times are GMT -8. The time now is 03:14 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.