possible reasons for error
 

I'm a new user of PhpDig and I'm using it to spider a very large website (over 300,000 pages, although many are dynamically generated). It seems to be working great (if a little slow - forking would be very useful).

However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table.

What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting.

Many thanks.

Hi. In the spider.php file look for the following two comments:

//mark the tempspider reccord as error

//insert in temp table for next level

Near these comments you will find if/else evaluations and the queries that set error to one.