Hi. I haven't tested the below code, but it should get around the following case:
Quote:
Originally posted by renehaentjens
With the following robots.txt, no indexing, I always get: links found: 0, ... Was recently indexed:
User-agent: phpdig
Disallow:
User-agent: *
Disallow: /
After removing this robots.txt, all goes fine.
My intention was to allow PhpDig to index, but tell the others to go away. Did I get the syntax wrong?
|
In robot_functions.php find the phpdigReadRobotsTxt function and in this function find:
PHP Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*(/([a-z0-9_/*+%.-]*))',$line,$regs))
{
if ($regs[1] == '/')
$exclude[$user_agent]['@ALL@'] = 1;
else
{
$exclude[$user_agent][str_replace('*','.*',str_replace('+','\\+',str_replace('.','\\.',$regs[2])))] = 1;
}
}
and replace with the following:
PHP Code:
if (eregi('[[:blank:]]*disallow:[[:blank:]]*(/([a-z0-9_/*+%.-]*))',$line,$regs))
{
if ($regs[1] == '/')
$exclude[$user_agent]['@ALL@'] = 1;
else
{
$exclude[$user_agent][str_replace('*','.*',str_replace('+','\\+',str_replace('.','\\.',$regs[2])))] = 1;
}
}
elseif (($user_agent == 'phpdig') && (eregi('[[:blank:]]*disallow:[[:blank:]]*',$line,$regs)))
{
$exclude['@NONE@'] = 1;
return $exclude;
}
Remember to remove any "word" wrapping in the above code.