Another tale of indexing woe

llamastephen · 07-15-2005, 04:20 AM

I'm trying to get search working on http://www.red-tape.org. There are around 80 pages in total on the site, most being delivered from mySQL.

I'm having trouble spidering the whole site (using the sitemap as the URI). The most I've managed to get to is 40 or so pages. I've just tried deleting and starting again and this time I just get to 21 pages (though there are 41 in the temp table).

I've searched through this forum and followed all the advice I can (PHPDIG_IN_DOMAIN = true etc) but I've run out of ideas.

The other thing is that the spider always ends up locked. I have tried stopping the spider, unlocking and repeating but always with the same result. How long should an 80 page site typically take? (I have tried in Firefox and IE6).

Any help or suggestions could save my sanity

thanks

Stephen

Charter · 07-18-2005, 04:37 PM

Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process.

07-15-2005, 04:20 AM	#1
llamastephen Green Mole Join Date: Jul 2005 Posts: 3	Another tale of indexing woe I'm trying to get search working on http://www.red-tape.org. There are around 80 pages in total on the site, most being delivered from mySQL. I'm having trouble spidering the whole site (using the sitemap as the URI). The most I've managed to get to is 40 or so pages. I've just tried deleting and starting again and this time I just get to 21 pages (though there are 41 in the temp table). I've searched through this forum and followed all the advice I can (PHPDIG_IN_DOMAIN = true etc) but I've run out of ideas. The other thing is that the spider always ends up locked. I have tried stopping the spider, unlocking and repeating but always with the same result. How long should an 80 page site typically take? (I have tried in Firefox and IE6). Any help or suggestions could save my sanity thanks Stephen

07-18-2005, 04:37 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Once spidering is complete, the tempspider table should be empty. Check whether a server timeout issue is happening or whether the host is killing the spidering process. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.