PhpDig.net - View Single Post - QUESTION: How-to Spider Multiple URL's, not just one at a time.

misterbearcom · 06-13-2004, 08:31 PM

I was wondering if you simply could simply create several copies of phpDig and run them simultaneously as separate spiders, but before you start digging to assign the site_ids and spider_ids to numbers that will not overlap each other.

So, if you had database one with the following:

site_id starts at 1
spider_id starts at 1

Then database 2:

site_id starts at 10,000,000.
spider_id starts at 10,000,000.

etc. until you've made as many spiders as you wanted.

Then run them all simultaneously.

Afterwards, dump the info for all of these databases into one final version of the phpdig database and transfer all the txt_content .txt files into the final site folder.

Would that work? I haven't personally tried it so I guess I should try it before suggesting it. In anycase, hopefully there will be further modifications to PHPDIG in the future which will remove this issue.

(My apologies if I sound ignorant. I'm the first one who will probably admit that I am.

)

06-13-2004, 08:31 PM	#4
misterbearcom Green Mole Join Date: Apr 2004 Location: Cali Posts: 10	Would this work? I was wondering if you simply could simply create several copies of phpDig and run them simultaneously as separate spiders, but before you start digging to assign the site_ids and spider_ids to numbers that will not overlap each other. So, if you had database one with the following: site_id starts at 1 spider_id starts at 1 Then database 2: site_id starts at 10,000,000. spider_id starts at 10,000,000. etc. until you've made as many spiders as you wanted. Then run them all simultaneously. Afterwards, dump the info for all of these databases into one final version of the phpdig database and transfer all the txt_content .txt files into the final site folder. Would that work? I haven't personally tried it so I guess I should try it before suggesting it. In anycase, hopefully there will be further modifications to PHPDIG in the future which will remove this issue. (My apologies if I sound ignorant. I'm the first one who will probably admit that I am. )