|
Well if a web site owner has excluded Phpdig from indexing then we should obey there wishes, other wise it wouldn't be fair to just put the "Nosebag" into anyones server and just dig away.
If everyone started to just dig into hosts then all spiders would end up being banned or excluded.
|