Thread
:
multiple crawlers
View Single Post
01-06-2005, 06:18 PM
#
6
jmitchell
Orange Mole
Join Date: Dec 2004
Location: Tennessee
Posts: 60
I guess I"m not totally understanding how this works...
If I'm correct, you install this script, and run it via cron jobs, and it will see if there are sites to be indexed, and add multiple spiders to handle it - right?
jmitchell
__________________
60,000 pages indexed!!!!!
http://www.sharemylink.com
jmitchell
View Public Profile
Find More Posts by jmitchell