[Carl Mikkelsen]

Attached is a patch file which could be applied to the robot_functions.php file includes with the phpdig distribution this morning.

This includes also changes to handle MSPOWERPOINT (which should be matched with declarations in includes/config.php), as well as some fixes inherited from an alternate robot_functions.php I downloaded.

There is one change where the DOMAIN field in cookies was causing php to crash. I removed the DOMAIN processing (without understanding the intent) which could cause problems. You can back out that change.

The main change is to move the robots.txt processing earlier, so that the http HEAD request is not performed.

I also fixed what seemed to this php novice to be problems escaping some characters in the robots.txt parsing. With this change, phpdig accepts "*" as a meta-character in robots.txt, allowing entries such as:
Disallow: dynamic-content/view*parm=

As I am unfamiliar with php, I'm asking both for php-related feedback, and for comments related to the intent of the changes.

Thanks,

-- Carl