|
I wrote the robots.txt file myself and i can remove it.
But nothing changes wether it's here or not.
I have some other sites on this server and it looks like I can't spider any of them from this server although I can spider them from other servers even with a robots.txt file set for those sites.
|