Daily indexing using CRON? (PhpDig, MailMan, Pipermail)

[muppet]

Hi Everybody, thanks for reading this.

I'm trying to work out the best way to re-index my site on a daily basis with minimal CPU impact. The site in question runs a couple of email lists which get about two dozen messages daily, and these are saved via pipermail.

In the PhpDig control panel I've deleted (clicked on the red cross) those paths that I don't need to index at all, i.e. the listinfo pages from mailman, the settings pages, and the TXT versions of the archives.

When I run the indexing manually from the control panel (by clicking on the green tick) the total process of reindexing takes less than a minute, and new posts are actually found without problem. What I'd like to do is to set up a crontab to do this sort of thing for me on a daily basis, but when I follow the instructions given on here I still end up with a total spidering time of between 25 minutes and 6 hours! This is way too big, can anybody tell me what I'm doing wrong?

The pages I need to index are:
http://www.uk-bandits.co.uk/pipermail/social/
http://www.uk-bandits.co.uk/pipermail/tech/
http://www.uk-bandits.co.uk/pipermail/test-list/
... and so I've set up my crontab thus:
0 1 1-31/1 * * /usr/bin/php -f /usr/local/phpdig/admin/spider.php /usr/local/phpdig/admin/temp/cronlist.txt >> /usr/local/phpdig/admin/temp/spider.log
The file cronlist.txt contains:
http://www.uk-bandits.co.uk/pipermail/test-list/
http://www.uk-bandits.co.uk/pipermail/social/
http://www.uk-bandits.co.uk/pipermail/tech/
Does PhpDig know that the sites I'm passing in via cronlist.txt are the same as those I've set via the control panel, and therefore apply the same exclusions as I've already specified there?
I guess what I really need is a high-level overview of how the whole thing works and how the different areas tie together ... without spending weeks and weeks on this.
I'm sure I'm not the only one using PhpDig with MailMan and Pipermail, what do other people do in my position?

Many thanks in advance for your time - I really appreciate any help on this!

Muppet.

[Charter]

Try clicking on the 'noway' icon (circle with dash) to delete and exclude a branch from future index. When indexing from shell, try decreasing SPIDER_MAX_LIMIT and LINKS_MAX_LIMIT, and also try setting LIMIT_TO_DIRECTORY to true. There is also a mod here (http://www.phpdig.net/forum/showthread.php?t=1095) that may be of use.

define('SPIDER_MAX_LIMIT',20); // max (re)index search depth - used for shell and admin panel dropdown
define('RESPIDER_LIMIT',5); // max update search depth - only used for browser, not used for shell

define('LINKS_MAX_LIMIT',20); // max (re)index links per - used for shell and admin panel dropdown
define('RELINKS_LIMIT',5); // max update links per - only used for browser, not used for shell

define('LIMIT_TO_DIRECTORY',false); // limit index to given (sub)directory where (sub)directories of give (sub)directory are NOT indexed
// for limit to directory, URL format must either have file at end or ending slash at end
// e.g., http://www.domain.com/dirs/ (WITH ending slash) or http://www.domain.com/dirs/dirs/index.php

define('ALLOW_SUBDIRECTORIES',false); // limit index to given (sub)directory where (sub)directories of give (sub)directory are indexed
// if set to true, LIMIT_TO_DIRECTORY must also be set to true