Thanks for your help on my earlier question Charter. Well the spider is crawling away now and hopefully working. I am wondering how to interpret the logs that show up in the web page while I'm indexing. Here's an example:
So the first url is obviously the root site I started on in this instance (it's not the my main site where the search box will be). The + means it has discovered a link for the next level right? My question is what does "Ok for
http://etc." mean? It seems to have added those domains in my main index page but they are not appearing in the searches from the search box?
Hopefully this is going to work
The plan is to create a search database of biblical studies related sites (you can see my main search page at
deinde.org. Do you think PHPDIG can handle that much data? I already have in my admin page ~650 hosts!!