PhpDig.net - View Single Post - Restart spider and index urls in temptable

jerrywin5 · 03-24-2005, 06:34 PM

When indexing a site, there are times when the process is stopped for whatever reason. I use the spider via a crontab to reduce this risk somewhat. When the process stops on a large site, the temp table is left with URIs to index. Rather than unlocking the domain, clearing the temptable, and restarting the spider to indexing all the URIs in the site, I would like to have the spider continue indexing the URIs found in the temptable. How can I do this?

A little more info:
The spider process seems to stop after 5 hours on the shared server I am using. I have the delay set to 3 seconds. The site I am trying to index now has 3,000 pages. Page indexing averages about two minutes. Almost all the URIs are in the same directory.

03-24-2005, 06:34 PM	#1
jerrywin5 Orange Mole Join Date: Mar 2004 Posts: 48	Restart spider and index urls in temptable When indexing a site, there are times when the process is stopped for whatever reason. I use the spider via a crontab to reduce this risk somewhat. When the process stops on a large site, the temp table is left with URIs to index. Rather than unlocking the domain, clearing the temptable, and restarting the spider to indexing all the URIs in the site, I would like to have the spider continue indexing the URIs found in the temptable. How can I do this? A little more info: The spider process seems to stop after 5 hours on the shared server I am using. I have the delay set to 3 seconds. The site I am trying to index now has 3,000 pages. Page indexing averages about two minutes. Almost all the URIs are in the same directory.