PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 12-19-2003, 10:45 AM   #1
bloodjelly
Purple Mole
 
Join Date: Dec 2003
Posts: 106
Spidering through a script

Hi -

I'm trying to run the spider in the "background" through a php script, and I have this so far:

PHP Code:
$GLOBALS['limit'] = 2;
$GLOBALS['url'] = "http://www.website.com/";
include 
'../search/admin/spider.php'
This seems to run the spider fine for the appropriate website, and the website is entered into the MySQL database, but searching doesn't work on it until I run an update manually.

Is there a better way to run the spider without having to directly enter in the site to be spidered? Thanks!
bloodjelly is offline   Reply With Quote
Old 12-19-2003, 11:58 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. You can run PhpDig from shell. Just set the following to the wanted search depth in the config file, make a text file with the full URLs, one per line, and use the below command.
PHP Code:
define('SPIDER_MAX_LIMIT',2);           //max recurse levels in sipder
define('SPIDER_DEFAULT_LIMIT',2);       //default value
define('RESPIDER_LIMIT',2);             //recurse limit for update 
Code:
php -f [PHPDIG_DIR]/admin/spider.php [File containing an urls list]
More shell indexing options can be found here.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 12-19-2003, 05:37 PM   #3
bloodjelly
Purple Mole
 
Join Date: Dec 2003
Posts: 106
This will work well, but is there a way to do it through a PHP script? I basically want "Install.php" to do x, y and z, plus get phpDig to spider a site, all in one execution without any user input. Can't I just feed spider.php the information it needs to spider the site? Thanks for the quick reply.

durr...just remembered I can do it with exec (/usr/bin/php -f spider.php); works well!

Last edited by bloodjelly; 01-06-2004 at 04:26 PM.
bloodjelly is offline   Reply With Quote
Old 03-15-2004, 11:56 AM   #4
bloodjelly
Purple Mole
 
Join Date: Dec 2003
Posts: 106
OK - I used this command to spider:

PHP Code:
exec("/usr/bin/php -f /path/to/spider.php $site >> /dev/null &"); 
where $site = http://www.mysite.com/

This worked great until I recently upgraded to a newer version of PHP, and now the command doesn't produce any results. I know there's a way to turn on/off PHP as an executable, but I can't find it in PHP.INI and I'm not sure where to look. Thanks for helping out.
bloodjelly is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
New To Script glengall1 Feedback & News 1 09-08-2006 09:11 PM
how to conf. the script? se7en How-to Forum 0 03-16-2006 06:31 AM
New authentication script manfred Mod Submissions 0 04-20-2004 09:36 AM
where do i start for installing this script ekimbo Script Installation 1 03-24-2004 10:41 PM
Script running on its own? druesome Troubleshooting 1 10-26-2003 09:15 AM


All times are GMT -8. The time now is 08:25 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.