Spider stops before all pages are indexed
 

Hi,

I am trying to spider 3 sections of my site via a sitemap which I have put together for the purpose. The problem is that sometimes the spider will index all the pages but often it will stop before the end. There is no error message or completion message, just nothing. If I go back to the admin page it says *Locked* - This behaviour appears to be completely r****m because each time it stops at a different point.

The site is http://www.lightingdirectory.com and the site map is http://www.lightingdirectory.com/site-map (This is the URL I'm using for indexing).

Observation - All of the pages appear in the tempspider table every time. So it appears that the pages are being indexed but not spidered.

Any help would be appreciated. Thanks.

Did you manage to solve this? I am having very similar problems.

Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process.

I haven't solved it yet. But when I set up the next crawl I'll also increase the time out value of my apache server and see whether that helps.