PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   IPs, SEs, & UAs (http://www.phpdig.net/forum/forumdisplay.php?f=35)
-   -   Using PhpDig on this site (http://www.phpdig.net/forum/showthread.php?t=617)

Charter 03-05-2004 08:23 AM

Using PhpDig on this site
 
Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks.

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"


All times are GMT -8. The time now is 07:01 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.