PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 03-31-2004, 12:18 PM   #1
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
How to change?

I have a large site about 800 pages but everytime i index it stops about 70 pages in. I noticed while it was doing the index that it is using

admin/spider.php?site_id=2&mode=small

Is there a way to change the mode from small to something else so that i can index the whole site?
a3719 is offline   Reply With Quote
Old 04-01-2004, 12:03 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. Do you mean that when it stops it isn't completely done with the index, or do you mean that it only indexes 70 of the 800 pages?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-01-2004, 04:04 AM   #3
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
yea when it stops it not done with the index i have to go back in a restart it and tell it to index again and it will get about 40 more pages then stop again and i have to keep redoing this
a3719 is offline   Reply With Quote
Old 04-01-2004, 04:19 AM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. This sounds like an issue possibly related to timeout or memory. What do you get in your error logs?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-01-2004, 04:31 AM   #5
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
the error logs that i have access to do not show timeouts or memory issues they are very basic logs and i can not access the other logs cause my host does not have them setup
a3719 is offline   Reply With Quote
Old 04-01-2004, 04:34 AM   #6
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
when i go back and tell it to respider the root it will go through all the previous pages saying date unchanged which is fine and then it will start to add about 40 or so more pages after it checks all the ones it did before. I got it to about 400 pages so far and it runs for about 40 minutes so i dont think that its a timeout error.
a3719 is offline   Reply With Quote
Old 04-01-2004, 05:06 AM   #7
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. Do you notice whether or not it only stops when it encounters a large file? If so, there is some code in this thread that might help.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-01-2004, 05:08 AM   #8
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
all of my files are small they are basic web pages with text most are about 300k or less. i will try that code tho just to make sure.
a3719 is offline   Reply With Quote
Old 04-01-2004, 05:19 AM   #9
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. Could it be that your host has some sort of auto shutdown for runaway or resource intensive scripts?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-01-2004, 05:26 AM   #10
a3719
Green Mole
 
Join Date: Mar 2004
Posts: 6
it may be that cause thats what i am starting to think. so i guess i will just index the pages about 50 at a time. Once i get all the pages indexed it reindexes them all fine. Like right now i have gone though and reindexed the root enough times to get about 500 pages now and it goes though all the previuos pages without a problem so i guess once i get all the pages indexed everything should be ok when it does the automatic reindex every week.
a3719 is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Change POST to GET esconsult1 Mod Requests 0 11-09-2005 03:56 AM
Change sleep noel How-to Forum 1 10-27-2005 09:58 AM
Change Identity of Spider Kishin How-to Forum 1 07-18-2005 04:03 PM
how to change user-agent ? 123av Mod Requests 11 04-21-2004 04:29 AM


All times are GMT -8. The time now is 08:19 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.