I've been struggling with phpdig for quite some time, thinking I MUST be doing something wrong.
Some history: I made a website, which is restricted through apache's auth mechanism in httpd.conf, for friends and family some years ago. Last year, I decided I needed a search engine. After fiddling around with a few that didn't work so well, I came across phpdig. I loved it, but couldn't for the life of me get it to work. So, over the last year, when I had time and wasn't too busy with work and family, I would come back to trying to get it to work.
FINALLY, after scouring posts today something clicked and I thought I would share in case anyone was as stupid as I have been... possibly preventing them -- too late for me
-- from going bald with frustration!
What I found out was that phpdig was happily going along and not complaining when it couldn't get to the password-protected content of my site. It would just skip over it and LOOK like it had indexed it. Only, when I went to search things nothing was there!! I couldn't figure this out and it was driving me nuts!!
Finally, a post I saw today made me look in the access logs for apache and, lo and behold, I noticed that every time I ran the spider, I was getting 401 errors in the log for the content that was password protected.
To solve, I temporarily removed the password protection from the server, reloaded the httpd.conf, and let phpdig go to work. NOW it indexed everything and I'm happy as a clam.
I hope this helps someone else someday.
