Spider Crawl stopping too early

X-Warlock · 05-04-2005, 07:39 AM

I've got a fresh install of phpdig on my server, the site's links are all properly linked and should be totally accessible, and I've tried numerous combinations on the search depth and links per, even as far as modifying the config to allow me a search and links setting of 100 each. But so far the absolute best I have been able to manage to index is less than 40 pages, on a site that should be returning several hundred.

What's causing it to just stop, and is there any way I can fix this or get around it and have it index the site properly?

Charter · 05-04-2005, 07:44 AM

Set LIMIT_TO_DIRECTORY to false, set PHPDIG_IN_DOMAIN to true, set "search depth" to a large number, set "links per" to zero, choose the "no" option.

X-Warlock · 05-12-2005, 11:58 AM

Done all of that, I've even got search depth set to 1000. The thing stops at 51 each and every time and there are definately pages not being indexed. I've tried god knows how many variations here and still I cannot get past 51.

Charter · 05-12-2005, 12:17 PM

When it stops, do you see an "indexing complete" message onscreen?

X-Warlock · 05-13-2005, 10:27 AM

Nope, I do not... and I have tried running this in both IE, and my default browser Firefox.

Charter · 05-19-2005, 07:01 AM

Firefox and IE should both be able to handle more than 51 links shown onscreen. Is your PHP set with safe mode on or off? If safe mode is off, does your host have a script set to kill certain processes?

05-04-2005, 07:39 AM	#1
X-Warlock Green Mole Join Date: May 2005 Posts: 3	Spider Crawl stopping too early I've got a fresh install of phpdig on my server, the site's links are all properly linked and should be totally accessible, and I've tried numerous combinations on the search depth and links per, even as far as modifying the config to allow me a search and links setting of 100 each. But so far the absolute best I have been able to manage to index is less than 40 pages, on a site that should be returning several hundred. What's causing it to just stop, and is there any way I can fix this or get around it and have it index the site properly?

05-04-2005, 07:44 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Set LIMIT_TO_DIRECTORY to false, set PHPDIG_IN_DOMAIN to true, set "search depth" to a large number, set "links per" to zero, choose the "no" option. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

05-12-2005, 12:17 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	When it stops, do you see an "indexing complete" message onscreen? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

05-13-2005, 10:27 AM	#5
X-Warlock Green Mole Join Date: May 2005 Posts: 3	Nope, I do not... and I have tried running this in both IE, and my default browser Firefox. Last edited by X-Warlock; 05-13-2005 at 10:31 AM.

05-19-2005, 07:01 AM	#6
Charter Head Mole Join Date: May 2003 Posts: 2,539	Firefox and IE should both be able to handle more than 51 links shown onscreen. Is your PHP set with safe mode on or off? If safe mode is off, does your host have a script set to kill certain processes? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

05-12-2005, 11:58 AM	#3
X-Warlock Green Mole Join Date: May 2005 Posts: 3	Done all of that, I've even got search depth set to 1000. The thing stops at 51 each and every time and there are definately pages not being indexed. I've tried god knows how many variations here and still I cannot get past 51.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
never ending crawl	oehk	Troubleshooting	0	12-12-2006 05:34 AM
No results after the crawl	GunMuse	Script Installation	8	04-12-2005 05:57 PM
crawl my website only	hula	Troubleshooting	0	08-29-2004 03:37 AM
Logical bug and stopping spidering	Konstantine	Bug Tracker	0	03-14-2004 12:03 AM
"tempspider" table fills itself without stopping	laurentxav	Troubleshooting	1	01-21-2004 09:09 AM