Stop spidering site after using an amount of bandwidth

[Grefix]

This may seem like an odd question but when crawling a site (for instance http://www.xxx.com) is it possible to stop the spider after it has spidered a certain amount of the site's bandwidth?

I ask this because my site spiders sites hosted by a free webhost with a limited amount of bandwith. A few days ago the spider got hung on one of these and used about 56MB of it's bandwidth. You can imagine the owner of that site wasn'tvery happy with that.

[Charter]

Hi. The below is untested, but you might try making the following changes in the spider.php file. Of course, another alternative is to avoid crawling such sites or use a search depth of zero or one.

PHP Code:

$sum_of_tempfilesize = 0;
// Spidering ...
while(($level <= $limit) && ($sum_of_tempfilesize <= X)) {
// $tempfilesize is text filesize, not the actual page size
// set X to be the maximum number of bytes allowed
...
$sum_of_tempfilesize = $sum_of_tempfilesize + $tempfilesize;
//Retrieve meta-tags for this page
...
// clean the tempspider table
$query = "DELETE FROM ".PHPDIG_DB_PREFIX."tempspider WHERE site_id=$site_id";