PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 03-24-2005, 07:34 PM   #1
jerrywin5
Orange Mole
 
Join Date: Mar 2004
Posts: 48
Restart spider and index urls in temptable

When indexing a site, there are times when the process is stopped for whatever reason. I use the spider via a crontab to reduce this risk somewhat. When the process stops on a large site, the temp table is left with URIs to index. Rather than unlocking the domain, clearing the temptable, and restarting the spider to indexing all the URIs in the site, I would like to have the spider continue indexing the URIs found in the temptable. How can I do this?

A little more info:
The spider process seems to stop after 5 hours on the shared server I am using. I have the delay set to 3 seconds. The site I am trying to index now has 3,000 pages. Page indexing averages about two minutes. Almost all the URIs are in the same directory.
jerrywin5 is offline   Reply With Quote
Old 04-06-2005, 02:18 PM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Make sure the locked and stopped columns of the sites table contain all zeros, and then try crawling a fake 404 link to see if PhpDig will pick up the tempspider info and resume.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
I: PHPDIG can not index 2+ URLs.. ? PL_90 Script Installation 0 10-22-2007 08:51 AM
How can I restart spidering after crash? yapuka How-to Forum 12 05-19-2004 04:13 AM
phpdig seems to guess some urls and spider it manute Troubleshooting 7 04-29-2004 02:49 AM
Admin approval for spider to index external URLs jerrywin5 Mod Requests 0 03-29-2004 10:37 PM
why is a temptable so big? (phpdigtempspider) tapete Troubleshooting 1 12-20-2003 07:57 PM


All times are GMT -8. The time now is 01:16 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.