PhpDig.net - View Single Post

rayvd · 11-19-2003, 01:20 PM

What I ended up doing was breaking my list of URL's into 7, 8, 9 or 10 sublists and then starting a crawler for each of them.

Pseudo-threading!

Doesn't make any individual site crawl faster, but the whole gets completed quicker.

11-19-2003, 01:20 PM	#4
rayvd Green Mole Join Date: Oct 2003 Location: Mesa, AZ Posts: 15	What I ended up doing was breaking my list of URL's into 7, 8, 9 or 10 sublists and then starting a crawler for each of them. Pseudo-threading! Doesn't make any individual site crawl faster, but the whole gets completed quicker.