custom depth of search per site in cgi (including urllist)
 

Hi,

since I use phpdig to spider several sites I wanted it to be possible to give a custom search depth per site in a urllist when spidering with a cron job.

So I made a few modifications to spider.php and robot_functions.php.

Thought this might be usefull for some off you, so I like to share it with you :)

IMPORTANT: modifications are made for version 1.8.3!

You need to add an extra column (site_limit) is to the table sites in the phpdig database sites

ALTER TABLE PREFIXsites ADD site_limit SMALLINT( 6 ) ;
------------------------------------------------------------------------------

example of list.txt
not specifying the depth or setting the depth to 0 will cause the spider to use the default depth as set in config.php

shell command:

site1.com will be spidered with a depth of 2
site2.com and site3.com will use the default value as specified in the config.php file
site4.com will be spidered with a depth of 2

It can also be used to spider a single site

shell command:examples: this will spider with a depth of 5If not specified or 0, then the default depth will be used as set up in config.php