For Those With Password Protected Sites

[bbenson]

I've been struggling with phpdig for quite some time, thinking I MUST be doing something wrong.

Some history: I made a website, which is restricted through apache's auth mechanism in httpd.conf, for friends and family some years ago. Last year, I decided I needed a search engine. After fiddling around with a few that didn't work so well, I came across phpdig. I loved it, but couldn't for the life of me get it to work. So, over the last year, when I had time and wasn't too busy with work and family, I would come back to trying to get it to work.

FINALLY, after scouring posts today something clicked and I thought I would share in case anyone was as stupid as I have been... possibly preventing them -- too late for me :) -- from going bald with frustration!

What I found out was that phpdig was happily going along and not complaining when it couldn't get to the password-protected content of my site. It would just skip over it and LOOK like it had indexed it. Only, when I went to search things nothing was there!! I couldn't figure this out and it was driving me nuts!! :rant:

Finally, a post I saw today made me look in the access logs for apache and, lo and behold, I noticed that every time I ran the spider, I was getting 401 errors in the log for the content that was password protected.

To solve, I temporarily removed the password protection from the server, reloaded the httpd.conf, and let phpdig go to work. NOW it indexed everything and I'm happy as a clam.

I hope this helps someone else someday. :D

[vinyl-junkie]

Hi, bbenson. Welcome to the forums! :D

That's certainly one way of getting around the authentication problem. However, your site might be vulnerable to unauthorized activity while you're indexing like this.

Have you seen this thread (http://www.phpdig.net/forum/showthread.php?t=1110&highlight=password)? The processes described there would provide the means to index your site, while keeping phpdig security in place.

Hope this helps. :)

[bbenson]

Hi Vinyl-Junkie. Thanks for the welcome. :D

Thanks for the info. My site, however, is a personal site that is pretty small -- and not widely available: no DNS, just hanging off my cable modem :smoke:. My removing the security while I index isn't a big deal... as I have it set, it only takes about 6 minutes to index what I want of the site. :)

I am very impressed with this software and many thanks to the forums, I have gotten many answers here over the last year or so... I'm just shy about posting and showing my ignorance. :D

Thanks again!!

[vinyl-junkie]

I have gotten many answers here over the last year or so... I'm just shy about posting and showing my ignorance. :DPlease don't ever be shy about asking a question here. If you haven't been able to find an answer to a question in the archives, it just might be something that not only you but a few other people would like to know the answer to. :)