Calling all persons spidering multiple domains

[Slider]

I would like to hear from different people using PhpDig that use it to crawl many different domains.

1. How many sites maximum have you spidered with PhpDig?
My curiosity is a result of wanting to have 10,000 sites listed at max and need to know if PhpDig can handle this or even what problems I might run into.

2. At your most sites crawled, how long does it take for crawling to finish?

3. Do you find that you run into any problems with non-relevant results and have to work to refine searches?

4. Any additonal information about problems that I may incur would be helpful as well.

I thank you in advance for your time and contributions to this thread.
David

[AllKnightAccess]

I am interested in this information too.

[Dave A]

Judging by my results a lot would depend on how deep you spider the other domains because the MSQL file on your host would grow quite large and as an estimate, I would imagine that around 500 domains would need around 40mb of MYSQL space to store the data, plus a few megabytes of space for the files and that is based on spidering to a depth of around three, picking up say ten linked files per domain.
Spidering speed varies a lot depending on the quality of the host that your spidering and it can change because of the amount of data flying around the web.
In one of the other forums a guy is offering a prebuilt database so you could contact him, to find out the results he has got from using PHPDIG to spider multiple domains he may well have a good idea what kind of sizes and storage space you may need and PHPDIG's speed over heaps of domains.
PHPDIG as software really is brilliant and does what it is supposed to do and the help and support offered in these forums is brilliant.
I hope that this helps you..
Many regards
Dave A