|
But what if some sites have restricted phpdig in their robots.txt file, due to other webmasters abusing the bot? Then everyone that uses phpdig is blocked, when it could just be one particular instance that was intended to be blocked. There is no way to distiguish between one site's use of phpdig over another.
And, just because I am asking this question doesn't mean I am planning some malicious agenda. There are so many sites that use phpdig, and I wouldn't want there misuse, if any, to reflect my own use.
|