[paullind]

review:

Trying to use shell scripting/crontab to call spider and make it spider list of websites. Apache mod php

Have set up correct crontab command, it calls spider.php and gives it the file with the list of websites(only one there now)

In spider.php $argv have values as:
the key is 0 the value is /path/to/phpdig/admin/spider.php
the key is 1 the value is /path/to/phpdig/admin/cronlist2.txt
$argc is 2

Spider.php calls config.php around line 82 and the script does not make it any further beyound this include statment to config.php.

Inside config.php at line 16 I believe this 'if' statement terminates the spidering process:
--------------------
if ((isset($relative_script_path)) && ($relative_script_path != ".") && ($relative_script_path != "..")) {
exit();
}
if (eregi("config.php",$_SERVER['SCRIPT_FILENAME']) || eregi("config.php",$_SERVER['REQUEST_URI'])) {
exit();
}
---------------------
My $relative_script_path is: /path/to/phpdig/ ,so it will exit in the first 'if'.

Why exit here? Should my $relative_script_path be something different?

Has anyone ever combined all the include files into one massive spider.php and run it to avoid potential errors with include files?

Thx again