Not spidering mod rewritten pages

wineo · 02-22-2006, 07:38 PM

I have two sub domains for my site and phpdig spiders the one nicely (including mod rewrites). BUT, it is not spidering further than this new page on the other sub domain. It will not go deeper than the page http://www.wineography.com/wine_reviews.htm . Here is the report:

Code:

SITE : http://www.wineography.com/
Exclude paths :
- @NONE@
- cellar_wine_search.htm
- member_details.htm
Duplicate of an existing document
1:http://www.wineography.com/wine_reviews.htm
(time : 00:00:05)
No link in temporary table
links found : 1
http://www.wineography.com/wine_reviews.htm
Optimizing tables...
Indexing complete !

Any ideas on how to force it?!

wineo · 02-22-2006, 07:53 PM

I physically indexed one of the mod pages () and it managed to index it and found one other, but still did not continue further. I had the link depth set to 20 and the limit set to 0...

SITE : http://www.wineography.com/
Exclude paths :
- @NONE@
- cellar_wine_search.htm
- member_details.htm
1:http://www.wineography.com/USA/country/212
(time : 00:00:06)
+
level 1...
2:http://www.wineography.com/top_wineries.htm
(time : 00:00:19)

No link in temporary table
links found : 2
http://www.wineography.com/USA/country/212
http://www.wineography.com/top_wineries.htm
Optimizing tables...
Indexing complete !

02-22-2006, 07:38 PM	#1
wineo Green Mole Join Date: Feb 2006 Posts: 2	Not spidering mod rewritten pages I have two sub domains for my site and phpdig spiders the one nicely (including mod rewrites). BUT, it is not spidering further than this new page on the other sub domain. It will not go deeper than the page http://www.wineography.com/wine_reviews.htm . Here is the report: Code: SITE : http://www.wineography.com/ Exclude paths : - @NONE@ - cellar_wine_search.htm - member_details.htm Duplicate of an existing document 1:http://www.wineography.com/wine_reviews.htm (time : 00:00:05) No link in temporary table links found : 1 http://www.wineography.com/wine_reviews.htm Optimizing tables... Indexing complete ! Any ideas on how to force it?!

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Mod-rewrite = spidering / URL problems	jcnorris	Troubleshooting	1	10-26-2006 09:38 AM
Spidering Stops, Site is Locked after 7 Pages	nevsie	Troubleshooting	3	03-02-2005 03:52 PM
not spidering all pages (too many links on page?)	mirdin	Troubleshooting	2	09-01-2004 06:08 AM
Spidering stops after 6 level 1 pages	Steve	Troubleshooting	3	05-20-2004 04:56 PM
Restrict Spidering to new pages only	bushpilot	How-to Forum	1	05-01-2004 08:07 AM

02-22-2006, 07:53 PM	#2
wineo Green Mole Join Date: Feb 2006 Posts: 2	I physically indexed one of the mod pages () and it managed to index it and found one other, but still did not continue further. I had the link depth set to 20 and the limit set to 0... SITE : http://www.wineography.com/ Exclude paths : - @NONE@ - cellar_wine_search.htm - member_details.htm 1:http://www.wineography.com/USA/country/212 (time : 00:00:06) + level 1... 2:http://www.wineography.com/top_wineries.htm (time : 00:00:19) No link in temporary table links found : 2 http://www.wineography.com/USA/country/212 http://www.wineography.com/top_wineries.htm Optimizing tables... Indexing complete !