Hi,
since I use phpdig to spider several sites I wanted it to be possible to give a custom search depth per site in a urllist when spidering with a cron job.
So I made a few modifications to spider.php and robot_functions.php.
Thought this might be usefull for some off you, so I like to share it with you
IMPORTANT: modifications are made for version 1.8.3!
You need to add an extra column (
site_limit) is to the table
sites in the phpdig database sites
ALTER TABLE PREFIXsites ADD site_limit SMALLINT( 6 ) ;
------------------------------------------------------------------------------
example of list.txt
not specifying the depth or setting the depth to 0 will cause the spider to use the default depth as set in config.php
shell command:
Code:
#php -f [PHPDIG_DIR]/admin/spider.php list.txt
site1.com will be spidered with a depth of 2
site2.com and site3.com will use the default value as specified in the config.php file
site4.com will be spidered with a depth of 2
It can also be used to spider a single site
shell command:
examples:
this will spider with a depth of 5
If not specified or 0, then the default depth will be used as set up in config.php