View Single Post
Old 03-02-2004, 04:54 AM   #1
edkay
Green Mole
 
Join Date: Feb 2004
Posts: 1
robots.txt comments

At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments.

Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)

As a simple fix, I added the following additional check:

PHP Code:
while (list($id,$line) = each($robots))
            {
            
// Ignore comments
            
if (strpos(trim($line),"#") === 0)
                continue;
            if (
ereg('^user-agent.... 
I would recommend that this (or similar) is included in future releases.
edkay is offline   Reply With Quote