At present (1.8.0), the phpdigReadRobotsTxt function doesn't understand comments.
Robots.txt comments begin with a #. At present, these are ignored and so commented out lines are still interpretted (not good!!)
As a simple fix, I added the following additional check:
PHP Code:
while (list($id,$line) = each($robots))
{
// Ignore comments
if (strpos(trim($line),"#") === 0)
continue;
if (ereg('^user-agent....
I would recommend that this (or similar) is included in future releases.