Yep, that looks correct. Let's try echoing out some stuff.
In robot_functions.php, right before:
PHP Code:
foreach ($file_content as $num => $line) {
put the following:
PHP Code:
// start echo stuff
$ijk_cnt = 0;
foreach ($file_content as $num => $line) {
if (trim($line)) {
if ($content_type == 'HTML' && trim($line) == PHPDIG_EXCLUDE_COMMENT) {
echo "Content type is HTML and PhpDig exclude comment found.<br>";
}
elseif (trim($line) == PHPDIG_INCLUDE_COMMENT) {
echo "PhpDig include comment found.<br>";
}
else {
if ($ijk_cnt == 0) {
echo "Content type is: " . $content_type . ".<br>";
$ijk_cnt++;
}
}
}
else {
echo "Trim line is false.<br><br>";
}
}
exit();
// end echo stuff
Now try to crawl a demo page with PhpDig exclude and include comments. What are your results for this?