PDA

View Full Version : robots.txt comments


edkay
03-02-2004, 04:54 AM
At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments.

Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)

As a simple fix, I added the following additional check:

while (list($id,$line) = each($robots))
{
// Ignore comments
if (strpos(trim($line),"#") === 0)
continue;
if (ereg('^user-agent....

I would recommend that this (or similar) is included in future releases.

marb
03-11-2004, 09:14 AM
In which file and where can I place this code?
I think it is robot_functions.php but I'm not sure.


Marten :)

Konstantine
03-12-2004, 12:41 PM
Yes, it's robot_functions.php. Hope, it will be in new version of phpdig :)