PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 05-04-2005, 08:39 AM   #1
X-Warlock
Green Mole
 
Join Date: May 2005
Posts: 3
Spider Crawl stopping too early

I've got a fresh install of phpdig on my server, the site's links are all properly linked and should be totally accessible, and I've tried numerous combinations on the search depth and links per, even as far as modifying the config to allow me a search and links setting of 100 each. But so far the absolute best I have been able to manage to index is less than 40 pages, on a site that should be returning several hundred.

What's causing it to just stop, and is there any way I can fix this or get around it and have it index the site properly?
X-Warlock is offline   Reply With Quote
Old 05-04-2005, 08:44 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Set LIMIT_TO_DIRECTORY to false, set PHPDIG_IN_DOMAIN to true, set "search depth" to a large number, set "links per" to zero, choose the "no" option.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 05-12-2005, 12:58 PM   #3
X-Warlock
Green Mole
 
Join Date: May 2005
Posts: 3
Done all of that, I've even got search depth set to 1000. The thing stops at 51 each and every time and there are definately pages not being indexed. I've tried god knows how many variations here and still I cannot get past 51.
X-Warlock is offline   Reply With Quote
Old 05-12-2005, 01:17 PM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
When it stops, do you see an "indexing complete" message onscreen?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 05-13-2005, 11:27 AM   #5
X-Warlock
Green Mole
 
Join Date: May 2005
Posts: 3
Nope, I do not... and I have tried running this in both IE, and my default browser Firefox.

Last edited by X-Warlock; 05-13-2005 at 11:31 AM.
X-Warlock is offline   Reply With Quote
Old 05-19-2005, 08:01 AM   #6
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Firefox and IE should both be able to handle more than 51 links shown onscreen. Is your PHP set with safe mode on or off? If safe mode is off, does your host have a script set to kill certain processes?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
never ending crawl oehk Troubleshooting 0 12-12-2006 06:34 AM
No results after the crawl GunMuse Script Installation 8 04-12-2005 06:57 PM
crawl my website only hula Troubleshooting 0 08-29-2004 04:37 AM
Logical bug and stopping spidering Konstantine Bug Tracker 0 03-14-2004 01:03 AM
"tempspider" table fills itself without stopping laurentxav Troubleshooting 1 01-21-2004 10:09 AM


All times are GMT -8. The time now is 12:42 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2023, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.