lighthouse
03-30-2004, 01:50 AM
I want to setup cron jobs to do the following:
1) spider a set of URLS every 7 days to a search depth of 2
2) spider a set of URLS every 28 days to a search depth of 5
Can this be done?... Can you specify the search depth paramater in the cron job?...
Also - can I spider 1 particular page of a website (the news page) without spidering the rest of the site (I'm pretty sure this is possible, but just want to confirm!!)
Thanks in advance
Gary
1) spider a set of URLS every 7 days to a search depth of 2
2) spider a set of URLS every 28 days to a search depth of 5
Can this be done?... Can you specify the search depth paramater in the cron job?...
Also - can I spider 1 particular page of a website (the news page) without spidering the rest of the site (I'm pretty sure this is possible, but just want to confirm!!)
Thanks in advance
Gary