|
03-01-2004, 08:16 AM | #1 |
Green Mole
Join Date: Feb 2004
Posts: 1
|
possible reasons for error
I'm a new user of PhpDig and I'm using it to spider a very large website (over 300,000 pages, although many are dynamically generated). It seems to be working great (if a little slow - forking would be very useful).
However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table. What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting. Many thanks. |
03-01-2004, 08:55 AM | #2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. In the spider.php file look for the following two comments:
//mark the tempspider reccord as error //insert in temp table for next level Near these comments you will find if/else evaluations and the queries that set error to one.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
Thread Tools | |
|
|