anyone think this is a good way to get phpDig to run multiple crawlers from sites in the database? When I spider.php I noticed that it would only do one site at a time and when it came across a site such as dmoz.org it would take days, if not weeks to index all of that.
http://rbhs.ath.cx/~reza/phpdig/wrapper.php