edkay
03-02-2004, 04:54 AM
At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments.
Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)
As a simple fix, I added the following additional check:
while (list($id,$line) = each($robots))
{
// Ignore comments
if (strpos(trim($line),"#") === 0)
continue;
if (ereg('^user-agent....
I would recommend that this (or similar) is included in future releases.
Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)
As a simple fix, I added the following additional check:
while (list($id,$line) = each($robots))
{
// Ignore comments
if (strpos(trim($line),"#") === 0)
continue;
if (ereg('^user-agent....
I would recommend that this (or similar) is included in future releases.