Thread: robots.txt
View Single Post
Old 12-05-2003, 02:40 PM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. I haven't tested the below code, but it should get around the following case:

Quote:
Originally posted by renehaentjens
With the following robots.txt, no indexing, I always get: links found: 0, ... Was recently indexed:

User-agent: phpdig
Disallow:

User-agent: *
Disallow: /

After removing this robots.txt, all goes fine.

My intention was to allow PhpDig to index, but tell the others to go away. Did I get the syntax wrong?

In robot_functions.php find the phpdigReadRobotsTxt function and in this function find:
PHP Code:
            if (eregi('[[:blank:]]*disallow:[[:blank:]]*(/([a-z0-9_/*+%.-]*))',$line,$regs))
                {
                if (
$regs[1] == '/')
                     
$exclude[$user_agent]['@ALL@'] = 1;
                else
                     {
                     
$exclude[$user_agent][str_replace('*','.*',str_replace('+','\\+',str_replace('.','\\.',$regs[2])))] = 1;
                     }
                } 
and replace with the following:
PHP Code:
            if (eregi('[[:blank:]]*disallow:[[:blank:]]*(/([a-z0-9_/*+%.-]*))',$line,$regs))
                {
                if (
$regs[1] == '/')
                     
$exclude[$user_agent]['@ALL@'] = 1;
                else
                     {
                     
$exclude[$user_agent][str_replace('*','.*',str_replace('+','\\+',str_replace('.','\\.',$regs[2])))] = 1;
                     }
                }
            elseif ((
$user_agent == 'phpdig') && (eregi('[[:blank:]]*disallow:[[:blank:]]*',$line,$regs)))
                {
                
$exclude['@NONE@'] = 1;
                return 
$exclude;
                } 
Remember to remove any "word" wrapping in the above code.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote