Understanding logs from web indexing
 

Thanks for your help on my earlier question Charter. Well the spider is crawling away now and hopefully working. I am wondering how to interpret the logs that show up in the web page while I'm indexing. Here's an example:

So the first url is obviously the root site I started on in this instance (it's not the my main site where the search box will be). The + means it has discovered a link for the next level right? My question is what does "Ok for http://etc." mean? It seems to have added those domains in my main index page but they are not appearing in the searches from the search box?

Hopefully this is going to work:) The plan is to create a search database of biblical studies related sites (you can see my main search page at
deinde.org. Do you think PHPDIG can handle that much data? I already have in my admin page ~650 hosts!!

Hi. The "+" means that PhpDig is indexing a page. The "OK for" means that PhpDig found a new site. The process is basically index site to search depth, index next site to search depth, index next site to search depth, etcetera, and as you are letting PhpDig crawl across domains, each new site gets added to an array and then indexed when the time comes. As for handling data, you may find this thread useful.