PDA

View Full Version : Limit number of spidered pages


Not Logged In
12-13-2003, 06:06 PM
I would like to put a limit on the number of pages that are indexed, save on some space. LIMIT 15?

thanks,
Robert

Charter
12-14-2003, 03:27 PM
Hi. The below has not been tested. When I get a chance I can test it, but for now it gives you something to try. ;)

In spider.php find the following:

// Spidering ...
while($level <= $limit) {

and replace with the following:

// Spidering ...
$my_spider_limit = 0;
while(($level <= $limit) && ($my_spider_limit < 15)) {
$my_spider_limit++;

Not Logged In
12-15-2003, 05:51 PM
Sorry to say that didn't work :bang:


I've tried to limit the number of times the tempspider is written but can't figure it out.


Any ideas?

Thanks for the help!

Charter
12-16-2003, 11:18 AM
Hi. Please undo the above changes and then see if this (http://www.phpdig.net/showthread.php?threadid=300) thread helps.

fr :: anonymus
12-16-2003, 03:10 PM
Charter : A bug in you code =>

// Spidering ...
$my_spider_limit = 0;
while(($level <= $limit) && ($my_spider_limit < 15)) {
$my_spider_limit++;

You must change && by OR :

// Spidering ...
$my_spider_limit = 0;
while(($level <= $limit) OR ($my_spider_limit < 15)) {
$my_spider_limit++;

Spider must stop when
$level <= $limit
OR when
$my_spider_limit <15

Isn't it ?

Charter
12-16-2003, 04:03 PM
Originally posted by fr :: anonymus
Charter : A bug in you code =>

// Spidering ...
$my_spider_limit = 0;
while(($level <= $limit) && ($my_spider_limit < 15)) {
$my_spider_limit++;

You must change && by OR :

// Spidering ...
$my_spider_limit = 0;
while(($level <= $limit) OR ($my_spider_limit < 15)) {
$my_spider_limit++;

Spider must stop when
$level <= $limit
OR when
$my_spider_limit <15

Isn't it ?

Hi. Thanks, but in that code I wanted the 'and' in place.

It was meant as when $level <= $limit and when $my_spider_limit < 15 do the loop, but it was the wrong piece of code to edit because all it does is basically limit the seach depth to fourteen without affecting the number of links found in a page. Silly me.

Please see this (http://www.phpdig.net/showthread.php?threadid=300) thread instead.