PhpDig.net - View Single Post

Charter · 12-16-2003, 10:14 AM

Hi. I haven't tested the below but what it should do is limit the number of links found per page to a max of 20, where each indexed page will only have a max of 20 links to follow. This is a per page rather than per site adjustment, so if you want to have a max of 100 links for one site, then you'll need to adjust the below added line and/or set a different search depth level.

In spider.php find the following:

PHP Code:


			
if (isset($urls) && is_array($urls)) {

and right after it place the following:

PHP Code:


			
$my_spider_limit = 20;

if(count($urls) > $my_spider_limit) {

   $urls = array_slice($urls, 0, $my_spider_limit);

}

You might be able to achieve similar results without making the above change by setting the search depth level to one. When the search depth level is one, only the page and links from that page are indexed. Of course this depends on how many links are in the page, so if you use the above code, you should be able to limit the links found on any given page to the first $my_spider_limit links.

12-16-2003, 10:14 AM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. I haven't tested the below but what it should do is limit the number of links found per page to a max of 20, where each indexed page will only have a max of 20 links to follow. This is a per page rather than per site adjustment, so if you want to have a max of 100 links for one site, then you'll need to adjust the below added line and/or set a different search depth level. In spider.php find the following: PHP Code: `if (isset($urls) && is_array($urls)) {` and right after it place the following: PHP Code: `$my_spider_limit = 20; if(count($urls) > $my_spider_limit) { $urls = array_slice($urls, 0, $my_spider_limit); }` You might be able to achieve similar results without making the above change by setting the search depth level to one. When the search depth level is one, only the page and links from that page are indexed. Of course this depends on how many links are in the page, so if you use the above code, you should be able to limit the links found on any given page to the first $my_spider_limit links. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.