I tried this, but unfortunately also without the result I hoped for. Setting the spider depth to 1 causes only my index page and the links found there to be spidered (which is only 5% of the entire site, the rest of the links are found in the next two levels in the site's link tree structure).
Increasing the spider depth to 2 allowed more of my site to be spidered, and more important: no files like I mentioned in the original posting were found. But also still a number of sub-pages weren't being spidered.
Increasing the spider depth one step further to 3 results in the entire site being spidered, but also in indexing all the files in the subdirectories involved which are no part of the link tree.
Seems to be a bug in PhpDig?
__________________
--
Life is wasted on the living
|