possible reasons for error

hennagaijin · 03-01-2004, 08:16 AM

I'm a new user of PhpDig and I'm using it to spider a very large website (over 300,000 pages, although many are dynamically generated). It seems to be working great (if a little slow - forking would be very useful).

However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table.

What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting.

Many thanks.

Charter · 03-01-2004, 08:55 AM

Hi. In the spider.php file look for the following two comments:

//mark the tempspider reccord as error

//insert in temp table for next level

Near these comments you will find if/else evaluations and the queries that set error to one.

03-01-2004, 08:16 AM	#1
hennagaijin Green Mole Join Date: Feb 2004 Posts: 1	possible reasons for error I'm a new user of PhpDig and I'm using it to spider a very large website (over 300,000 pages, although many are dynamically generated). It seems to be working great (if a little slow - forking would be very useful). However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table. What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting. Many thanks.

03-01-2004, 08:55 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. In the spider.php file look for the following two comments: //mark the tempspider reccord as error //insert in temp table for next level Near these comments you will find if/else evaluations and the queries that set error to one. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.