PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 01-14-2004, 06:58 AM   #1
Grefix
Green Mole
 
Join Date: Jan 2004
Posts: 1
Stop spidering site after using an amount of bandwidth

This may seem like an odd question but when crawling a site (for instance http://www.xxx.com) is it possible to stop the spider after it has spidered a certain amount of the site's bandwidth?

I ask this because my site spiders sites hosted by a free webhost with a limited amount of bandwith. A few days ago the spider got hung on one of these and used about 56MB of it's bandwidth. You can imagine the owner of that site wasn'tvery happy with that.
Grefix is offline   Reply With Quote
Old 01-14-2004, 10:45 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. The below is untested, but you might try making the following changes in the spider.php file. Of course, another alternative is to avoid crawling such sites or use a search depth of zero or one.
PHP Code:
$sum_of_tempfilesize 0;
// Spidering ...
while(($level <= $limit) && ($sum_of_tempfilesize <= X)) {
// $tempfilesize is text filesize, not the actual page size
// set X to be the maximum number of bytes allowed
...
$sum_of_tempfilesize $sum_of_tempfilesize $tempfilesize;
//Retrieve meta-tags for this page
...
// clean the tempspider table
$query "DELETE FROM ".PHPDIG_DB_PREFIX."tempspider WHERE site_id=$site_id"
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Bandwidth requirements? new2dev How-to Forum 1 02-17-2005 05:36 AM
phpdig blocked when spidering any site heli Troubleshooting 3 09-30-2004 10:42 AM
Fixing spider.php, protecting from locking site after timeout or users stop Konstantine Mod Submissions 3 04-09-2004 12:37 PM
Spidering issue with my site pager Troubleshooting 5 01-19-2004 10:05 AM
Problems spidering dynamic site Ph0nK Troubleshooting 1 01-13-2004 03:39 PM


All times are GMT -8. The time now is 01:48 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.