|
How long?
I have installed PhpDig on my winxp box running Apache .
I have installed a robots.txt in the root dir.
When trying to spider my site I get a list of all the directories which are off limits and the URL becomes locked.
How long will "Spidering" take ? Must be less than 500kb of html pages. Should this take minutes or longer?
Regards
Steve
|