Another tale of indexing woe

[llamastephen]

I'm trying to get search working on http://www.red-tape.org. There are around 80 pages in total on the site, most being delivered from mySQL.

I'm having trouble spidering the whole site (using the sitemap as the URI). The most I've managed to get to is 40 or so pages. I've just tried deleting and starting again and this time I just get to 21 pages (though there are 41 in the temp table).

I've searched through this forum and followed all the advice I can (PHPDIG_IN_DOMAIN = true etc) but I've run out of ideas.

The other thing is that the spider always ends up locked. I have tried stopping the spider, unlocking and repeating but always with the same result. How long should an 80 page site typically take? (I have tried in Firefox and IE6).

Any help or suggestions could save my sanity

thanks

Stephen

[Charter]

Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process.