PhpDig.net

Go Back   PhpDig.net > General Forums > IPs, SEs, & UAs

Reply
 
Thread Tools
Old 03-05-2004, 09:23 AM   #1
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Using PhpDig on this site

Hi. Most 'big player' robots are allowed to crawl this site. However, please do not use PhpDig to try and crawl this site. Any income currently earned from this site does not cover the costs to operate this site. Please be courteous of this fact, when using a 'homemade' robot like PhpDig, before crawling this or other sites. Thanks.

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:41 -0800] "HEAD /robots.txt HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:42 -0800] "GET /robots.txt HTTP/1.0" 200 1145 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:43 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

xxx.xxx.xxx.xxx - - [05/Mar/2004:08:28:44 -0800] "HEAD / HTTP/1.1" 200 0 "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New phpDig indexes whole site now or? JAB Creations The Mole Hole 0 11-07-2007 04:14 AM
Switch of site with PHPDig to new server leslief Script Installation 0 02-13-2007 09:19 AM
successful indexing of every site but site where phpdig is served phillystyle123 Troubleshooting 1 02-21-2005 10:06 PM
phpdig blocked when spidering any site heli Troubleshooting 3 09-30-2004 11:42 AM
Can't get PHPDig to index an htaccess protected site mlerch@mac.com Troubleshooting 28 02-25-2004 04:13 PM


All times are GMT -8. The time now is 12:22 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.