PhpDig.net - View Single Post - cannot spider own site

web newsroom · 05-23-2004, 03:32 PM

actually just figured it out. pretty simple too.

Seems that if I delete the robots.txt it will allow me to spider.

So ill go back and delete the directories I dont want indexed and then replace the tobots.txt

thanks

05-23-2004, 03:32 PM	#2
web newsroom Green Mole Join Date: May 2004 Posts: 4	actually just figured it out. pretty simple too. Seems that if I delete the robots.txt it will allow me to spider. So ill go back and delete the directories I dont want indexed and then replace the tobots.txt thanks