PhpDig.net - View Single Post - First crawl works, then subsequent crawls go on forever

andyt · 03-26-2006, 11:56 PM

Hi,

I've got phpDig working on one site and it's fantastic. But I'm having problems setting it up on a different site.

The first time I crawl the site it works no problem but the next time I try, either through the admin interface or the command line, the spider just goes on and on examining the pages. I've had it running for half an hour before I stopped it once. The first crawl only takes a few minutes.

The site I'm indexing has the same menu on every page and I'm setting the search depth to 2 and link per to zero (check all).

Can anyone suggest what the problem might be?

Thanks

Andy

03-26-2006, 11:56 PM	#1
andyt Green Mole Join Date: Mar 2006 Posts: 1	First crawl works, then subsequent crawls go on forever Hi, I've got phpDig working on one site and it's fantastic. But I'm having problems setting it up on a different site. The first time I crawl the site it works no problem but the next time I try, either through the admin interface or the command line, the spider just goes on and on examining the pages. I've had it running for half an hour before I stopped it once. The first crawl only takes a few minutes. The site I'm indexing has the same menu on every page and I'm setting the search depth to 2 and link per to zero (check all). Can anyone suggest what the problem might be? Thanks Andy