PDA

View Full Version : Different search depths for different cron jobs


lighthouse
03-30-2004, 02:50 AM
I want to setup cron jobs to do the following:

1) spider a set of URLS every 7 days to a search depth of 2
2) spider a set of URLS every 28 days to a search depth of 5

Can this be done?... Can you specify the search depth paramater in the cron job?...


Also - can I spider 1 particular page of a website (the news page) without spidering the rest of the site (I'm pretty sure this is possible, but just want to confirm!!)


Thanks in advance
Gary

Charter
03-30-2004, 09:15 AM
Hi. Perhaps try the following.

In spider.php find:

if ($argc > 1) {

and afterwards put:

$limit = argv[2];

Usage: php -f spider.php [option] [number]
where option and number must be specified.