When a very lot of links...

fr :: anonymus · 12-09-2003, 03:53 PM

Hi,

When there are a very lot of links (most than 3000) in the same page, spider record all, and scan them.
But it's very long between 2 links :

17:http://www.*****.net/docs/php/annote...properties.php
(temps : 00:49:41)

18:http://www..*****.net/docs/php/annot...es-by-list.php
(temps : 00:54:05)

19:http://www..*****.net/docs/php/annot...-by-regexp.php
(temps : 00:57:36)

and time, in (at) the end, is very important. (3-5 days ?)

Do you know how to make it faster, without mode localhost ?

Thanks, Anonymus.

fr :: anonymus · 12-09-2003, 03:54 PM

**** = nexen.

Charter · 12-10-2003, 07:51 AM

Hi. Are you indexing http://www.nexen.net/docs/php/annotee/indexes.functions.php at level one? If not already done, perhaps try to be selective when indexing a site by choosing the most relevant page to index at level one. Also, if available, perhaps try to index from shell rather than with the browser interface. If indexing from shell, the following can be set in the config file to limit the level to one.

PHP Code:


			
define('SPIDER_MAX_LIMIT',1);           //max recurse levels in sipder

define('SPIDER_DEFAULT_LIMIT',1);       //default value

define('RESPIDER_LIMIT',1);             //recurse limit for update

fr :: anonymus · 12-10-2003, 12:38 PM

That's for ..../php/annotee/indexes.functions.php

All are in level 1.

Ok, thanks.

Anonymus.

12-09-2003, 03:53 PM	#1
fr :: anonymus Green Mole Join Date: Dec 2003 Location: Lyon, France Posts: 17	When a very lot of links... Hi, When there are a very lot of links (most than 3000) in the same page, spider record all, and scan them. But it's very long between 2 links : 17:http://www.***.net/docs/php/annote...properties.php (temps : 00:49:41) 18:http://www..*.net/docs/php/annot...es-by-list.php (temps : 00:54:05) 19:http://www..***.net/docs/php/annot...-by-regexp.php (temps : 00:57:36) and time, in (at) the end, is very important. (3-5 days ?) Do you know how to make it faster, without mode localhost ? Thanks, Anonymus.

12-10-2003, 07:51 AM	#3
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Are you indexing http://www.nexen.net/docs/php/annotee/indexes.functions.php at level one? If not already done, perhaps try to be selective when indexing a site by choosing the most relevant page to index at level one. Also, if available, perhaps try to index from shell rather than with the browser interface. If indexing from shell, the following can be set in the config file to limit the level to one. PHP Code: `define('SPIDER_MAX_LIMIT',1); //max recurse levels in sipder define('SPIDER_DEFAULT_LIMIT',1); //default value define('RESPIDER_LIMIT',1); //recurse limit for update` __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Phpdig find a lot of 404 errors	Danydan	Troubleshooting	3	12-18-2005 01:11 AM

12-09-2003, 03:54 PM	#2
fr :: anonymus Green Mole Join Date: Dec 2003 Location: Lyon, France Posts: 17	**** = nexen.

12-10-2003, 12:38 PM	#4
fr :: anonymus Green Mole Join Date: Dec 2003 Location: Lyon, France Posts: 17	That's for ..../php/annotee/indexes.functions.php All are in level 1. Ok, thanks. Anonymus.