PhpDig.net - View Single Post

Charter · 03-05-2004, 08:23 AM

Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks.

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

03-05-2004, 08:23 AM	#1
Charter Head Mole Join Date: May 2003 Posts: 2,539	Using PhpDig on this site Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks. xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.