View Full Version : spider.php via bash

02-07-2004, 01:50 PM
hello together,

i'm trying to start the spider-process via shell - but nothing happens - so i read all articles concerning this in the forum.

but nothing works!

/usr/local/bin/php4 -f /phpdig/admin/spider.php http://www.domain.com >> phpdigg.log

when i take the -w switch(instead of -f) i see the code from spider.php => this tells me all paths are ok
several other scripts run without problems from shell and cron

but spider.php won't :-(

sending the command=>new prompt=>but nothing happens

please help :-)

02-07-2004, 01:55 PM
Hi. Try this and then let me know what's in the phpdigg.log file:

/usr/local/bin/php4 -f /phpdig/admin/spider.php http://www.domain.com > phpdigg.log 2>&1

02-07-2004, 02:02 PM
hello charter,

this is the fastes response i ever got - thanks

content of logfile:
X-Powered-By: PHP/4.2.3
Content-type: text/html

02-07-2004, 02:04 PM
Hi. What does the following produce?

CD to the admin directory and then do the following:

php -f spider.php http://www.domain.com > phpdigg.log 2>&1

02-07-2004, 02:11 PM
hi charter,

no i hope it will run for the next 7 hours and index all sites.

thanks in advance
tomas from germany

02-07-2004, 02:36 PM
but one question remains,

when calling the script with cron i need the full path
from root :-(

02-07-2004, 02:43 PM
Hi. If you are following this (http://www.phpdig.net/showthread.php?threadid=323) tutorial, and you place everything in the admin directory, then you can use the following to spider on the 1st and 15th of every month:

0 0 1,15 * * php -f spider.php cronlist.txt > phpdigg.log

02-07-2004, 02:49 PM
hi charter,

ok - i try this out - but why does it not
work with the full path - i do this with some
other php scripts and they work fine???

02-07-2004, 02:54 PM
Hi. Full path to PHP or other scripts?

0 0 1,15 * * /full/path/to/php -f /full/path/to/admin/spider.php cronlist.txt > phpdigg.log

What OS are you using?

02-07-2004, 03:00 PM
full path to other scripts
won't i loose all other cronjobs in my crontab
doing it like the tutorial?

02-07-2004, 03:11 PM
Hi. The command first posted was the following:

/usr/local/bin/php4 -f /phpdig/admin/spider.php http://www.domain.com >> phpdigg.log

Was /phpdig/admin/spider.php the full path to spider.php? Perhaps try /usr/local/bin/php or just php instead of /usr/local/bin/php4 in the command.

I'm not all that familiar with Debain, but with Linux you can set a number of cronjobs up to the limit imposed by a host. Perhaps try setting a test cronjob that prints some text to a file and see if it removes the PhpDig cronjob.

02-07-2004, 03:33 PM
yes - as i thought this overwrites the existing crontab from the logged user -
but charter - why all this ???
a lot of other scripts called by crontab work fine
why can't i call this from my user directory this way like all the others:
5 * * * 1,2,3,4,5 /usr/local/bin/php4 /homepages/22/d86961/htdocs/cron/rep_01.php > /dev/null

02-07-2004, 03:42 PM
Hi. How many other cronjobs do you have running? Maybe you are at your limit, but I'm guessing not. One thing is that the config.php file has the following code:

if ((isset($relative_script_path)) && ($relative_script_path != ".") && ($relative_script_path != "..")) {

This code was implemented for security reasons. If the $relative_script_path from shell is different than dot or double dot, then the process will exit. If this is the case, just add the relative path needed in the above code.

02-07-2004, 03:58 PM
gotcha -
this is the troublemaker:
doing this:
if ((isset($relative_script_path)) && ($relative_script_path != ".") && ($relative_script_path != "..")) {
if (eregi("config.php",$_SERVER['SCRIPT_FILENAME']) || eregi("config.php",$_SERVER['REQUEST_URI'])) {

now it runs

02-07-2004, 04:11 PM
Hi. Glad it's now working, but you don't want to comment that section out. It was added for security reasons, and unless you are running locally (no other person has access) then that is opening you up to a world of trouble! Rather you need to allow for the relative path in the code, the same as was done for dot and double dot.

02-07-2004, 04:16 PM
but what is the relative path to:
user dir: /homepages/22/d86961/htdocs/
config.php: /homepages/22/d86961/htdocs/www/en/phpdig/includes/config.php

($relative_script_path != "???")

02-07-2004, 04:23 PM
H. Just echo the following.

In spider.php right before the following:

// header of page

put the following:

echo $relative_script_path;

Add what is echoed to the code in the config.php file and then undo the above code.

Do not just comment out the code like was previously posted because otherwise badness may find you. :eek: