Charter,
Thanks for your prompt reply. I had already tried your advice based in other threads.
I just rechecked the settings in my config-file, and they were set as you recommended. I selected 20 as the depth and zero links per and set the default values option to no.
I tried to index the page
http://globalflyfisher.com/patterns/
The result was the usual:
-------------------------------
1:http://globalflyfisher.com/patterns/
(time : 00:00:01)
No link in temporary table
links found : 1
http://globalflyfisher.com/patterns/
Optimizing tables...
Indexing complete ! [Back] to admin interface.
-------------------------------
And that page has plenty links and no fancy JavaScript or anything else to provoke the crawler. My robots.txt does not exclude the page and if I try a r****m subpage which is linked from the page above, like
http://globalflyfisher.com/patterns/branchu/ the crawler willingly crawls the links and subpages on that page...!
-------------------------------
1:http://globalflyfisher.com/patterns/branchu/
(time : 00:00:01)
+ + + + +
level 1...
2:http://globalflyfisher.com/patterns/...7&caller=index
(time : 00:00:02)
3:http://globalflyfisher.com/patterns/...6&caller=index
(time : 00:00:03)
4:http://globalflyfisher.com/patterns/...5&caller=index
(time : 00:00:03)
5:http://globalflyfisher.com/patterns/branchu/all.php
(time : 00:00:04)
+ + + +
6:http://globalflyfisher.com/patterns/branchu/index.php
(time : 00:00:06)
level 2...
7:http://globalflyfisher.com/patterns/...516&caller=all
(time : 00:00:07)
8:http://globalflyfisher.com/patterns/...517&caller=all
(time : 00:00:08)
9:http://globalflyfisher.com/patterns/...515&caller=all
(time : 00:00:09)
10:http://globalflyfisher.com/patterns/...514&caller=all
(time : 00:00:09)
No link in temporary table
links found : 10
http://globalflyfisher.com/patterns/branchu/
http://globalflyfisher.com/patterns/...7&caller=index
http://globalflyfisher.com/patterns/...6&caller=index
http://globalflyfisher.com/patterns/...5&caller=index
http://globalflyfisher.com/patterns/branchu/all.php
http://globalflyfisher.com/patterns/branchu/index.php
http://globalflyfisher.com/patterns/...516&caller=all
http://globalflyfisher.com/patterns/...517&caller=all
http://globalflyfisher.com/patterns/...515&caller=all
http://globalflyfisher.com/patterns/...514&caller=all
Optimizing tables...
Indexing complete ! [Back] to admin interface.
-------------------------------
I can't see any pattern in this behaviour.
I use the phpdigExclude and phpdigInclude tags to exclude head and footer, and that seems to work OK.
Would it help me to clean out the whole search index and retry the process from scratch? I hate such solutions, which really don't explain anything, but I will try it if I can obtain the desired effect: a complete site indexing in one go.
Again: thanks for a great product. It does a very good job with the pages it has indexed.
Martin