hennagaijin
03-01-2004, 08:16 AM
I'm a new user of PhpDig and I'm using it to spider a very large website (over 300,000 pages, although many are dynamically generated). It seems to be working great (if a little slow - forking would be very useful).
However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table.
What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting.
Many thanks.
However, there's one page/directory that it just won't spider. It's a very basic page that is linked to from every other page on the site, but for some reason it keeps getting logged with error = 1 in the tempspider table, and it never makes it to the spider table.
What are the different possible reasons why a page would be added to tempspider with error = 1? I feel like if I had a list of possibilities I'd have more options for troubleshooting.
Many thanks.