This is strange. It finds the include, so things should get indexed, but it doesn't find the exclude. With exclude before include, and each on their own line, it seems that:
PHP Code:
if ($content_type == 'HTML' && trim($line) == PHPDIG_EXCLUDE_COMMENT) {
is coming out false, even though the content type is html and there is no typo. How about change:
PHP Code:
if ($content_type == 'HTML' && trim($line) == PHPDIG_EXCLUDE_COMMENT) {
to the following:
PHP Code:
if (trim($line) == PHPDIG_EXCLUDE_COMMENT) {
and try to crawl a demo page again.