PDA

View Full Version : Less hosts than I put


noel
11-06-2005, 09:36 AM
Hi CHARTER,

I put 773 urls and after the spider stops without problems, phpdig finds just 588 host. I don't understand why, all the urls works well, all the sites exists.

Thank you

Noël

Charter
11-06-2005, 01:08 PM
Maybe certain sites block spiders using a robots.txt or .htaccess file. Maybe the encoding could not be understood in the phpdigMakeUTF8 function. Maybe the status from the phpdigTestUrl function returned NOHOST, NOFILE, LOOP, or NEWHOST. Maybe something else.