Hi,
I have hard work for PhpDig - about milion domains to index
For tests I make environment on my PC (Athlon 1.8GHz, 1 Gigs RAM...fast ADSL net connection). I try soft on about 10 sites and I was surprised when looking at processor, disk and network usage - was minimal. Spidering was site by site and rather slow. Is it any way to run spider simultanously over not one but let say 5 sites (not from shell) - using script modyfication as parameter?
Marcin