PDA

View Full Version : Not spidering mod rewritten pages


wineo
02-22-2006, 07:38 PM
I have two sub domains for my site and phpdig spiders the one nicely (including mod rewrites). BUT, it is not spidering further than this new page on the other sub domain. It will not go deeper than the page http://www.wineography.com/wine_reviews.htm . Here is the report:


SITE : http://www.wineography.com/
Exclude paths :
- @NONE@
- cellar_wine_search.htm
- member_details.htm
Duplicate of an existing document
1:http://www.wineography.com/wine_reviews.htm
(time : 00:00:05)
No link in temporary table
links found : 1
http://www.wineography.com/wine_reviews.htm
Optimizing tables...
Indexing complete !


Any ideas on how to force it?!

wineo
02-22-2006, 07:53 PM
I physically indexed one of the mod pages () and it managed to index it and found one other, but still did not continue further. I had the link depth set to 20 and the limit set to 0...

SITE : http://www.wineography.com/
Exclude paths :
- @NONE@
- cellar_wine_search.htm
- member_details.htm
1:http://www.wineography.com/USA/country/212
(time : 00:00:06)
+
level 1...
2:http://www.wineography.com/top_wineries.htm
(time : 00:00:19)

No link in temporary table
links found : 2
http://www.wineography.com/USA/country/212
http://www.wineography.com/top_wineries.htm
Optimizing tables...
Indexing complete !