View Single Post
Old 03-30-2004, 01:50 AM   #1
lighthouse
Green Mole
 
Join Date: Mar 2004
Posts: 20
Different search depths for different cron jobs

I want to setup cron jobs to do the following:

1) spider a set of URLS every 7 days to a search depth of 2
2) spider a set of URLS every 28 days to a search depth of 5

Can this be done?... Can you specify the search depth paramater in the cron job?...


Also - can I spider 1 particular page of a website (the news page) without spidering the rest of the site (I'm pretty sure this is possible, but just want to confirm!!)


Thanks in advance
Gary
lighthouse is offline   Reply With Quote