Thanks again for the help.
I was able to spider the remaining directories, except for one that does not have an html link. I could just create the link temporarily if it is that important.
Thanks for the good utility! - it's nice to be off of the web-based, outsourced search.