PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 04-21-2004, 05:26 AM   #1
tryangle
Green Mole
 
Join Date: Jan 2004
Location: Penns Woods
Posts: 17
Multiple spiders

Hi,

Is it just a matter of opening another shell, or browser tab to run another instance of the spider... or is there more to it?

Tia for a response on this...
tryangle is offline   Reply With Quote
Old 04-22-2004, 11:09 AM   #2
marb
Green Mole
 
Join Date: Mar 2004
Posts: 19
I have install spider serveral times, in different dirs use one db.
Think this the fasted way to run more than one spider.
Have spider more than 19000 urls in 3 weeks, take 1 gb traffic.
More/to much spiders make the server slow.
Search at the same, during spidering with more than one spider at the same time, results show can come slow also.
I have experience, use more than 5 spider at the same time the script hang.
I could save the db, trough copy to a other db, but must reinstall the script and delete the old db.

Marten
marb is offline   Reply With Quote
Old 04-23-2004, 07:35 PM   #3
tryangle
Green Mole
 
Join Date: Jan 2004
Location: Penns Woods
Posts: 17
Smile Thank you marb...

By the way... did you figure out how the wrapper.php works? I read through the comments in the script, but I'm still not sure how to implement it for multiple spiders and url.txt files.
tryangle is offline   Reply With Quote
Old 04-24-2004, 03:43 AM   #4
marb
Green Mole
 
Join Date: Mar 2004
Posts: 19
tryangle wrote;
Quote:
By the way... did you figure out how the wrapper.php works? I read through the comments in the script, but I'm still not sure how to implement it for multiple spiders and url.txt files.
No I dont work with wrapper.php, I want to tryout.
The results I get of multiple spidering is coming from one DB.
No matter how may times or on wich url's I install the script.
I'm use not the text_content file, Have change this in the congi file.
Quote:
define('SUMMARY_LENGTH',50000); //length of results summary

define('TEXT_CONTENT_PATH','text_content/'); //Text content files path
define('CONTENT_TEXT',0); //Activates/deactivates the
//storage of text content.
It was a idea of René Haentjes see;
http://phpdig.net/showthread.php?s=&threadid=796
and that works good.

Marten
marb is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
wrapper/multiple spiders... CentaurAtlas Mod Submissions 5 11-19-2006 01:07 PM
Multiple Spiders jmitchell How-to Forum 3 12-16-2004 05:43 PM
Locked one of my spiders? megabaz Troubleshooting 4 10-23-2004 05:37 PM
1.8.3 spiders slow, 1.6.3 spiders same site fast Wayne McBryde Troubleshooting 0 09-21-2004 08:10 PM
2 or more Spiders firestarter Mod Requests 2 03-29-2004 12:59 PM


All times are GMT -8. The time now is 06:43 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.