Charter
03-05-2004, 08:23 AM
Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks.
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"