PhpDig.net - View Single Post - Spiders site first time, but update doesn't spider

Ensim · 11-23-2004, 08:17 AM

Thanks again Vinyl-Junkie. I gave up on the command line version, I've had problems with command line php before though, it doesn't appear to be the same exact php that runs in apache on our server.

For those interested, here is the command line I came up with that works, it uses wget to call the php via apache:

/usr/bin/wget --timeout 3600 --http-user your_htaccess_username --http-passwd your_htaccess_password "http://www.mysite.org/phpdig/admin/update.php?path=&site_id=1&exp=1"
1>/dev/null 2>/dev/null

(where it is all one line with no breaks)

Here's the explanation of the command:

/usr/bin/wget
Replace this with what ever path there is to your wget, find out using "which wget" at the SSH/telnet prompt.

--timeout 3600
this is the timeout in seconds. I removed the sleep(5) in the spider.php, if you keep it there increase this timeout a lot, my update.php call takes 20 seconds so an hour should be plenty for me. I also removed all the "set_timeout_limit" calls from PHPDig since my PHP safe mode gave errors that resulted in "update form" not reindexing the site.

--http-user & --http_passwd
I turned off the PHPDig admin user/pass by updating this in config.php:
define('PHPDIG_ADM_AUTH','0'); Then I password protected the admin directory using apache .htaccess files, and included their username and password as these parameters.

"http://www.mysite.org/phpdig/admin/update.php?path=&site_id=1&exp=1"
This link should work in your browser (after logging in) and force a site reindex. I got this link by selecting my site in the admin and clicking "update form", then copying the link that is there on the "root" green checkbox. I had to include the quotes so wget understood the full link.

1>/dev/null 2>/dev/null
This is optional. Without it, each time this command runs wget will save another copy of the page loaded which you may want as a record of the indexing. By including this, no files are created by wget.

Then you can add this to your cronjob so it can reindex automatically. Hopefully this will help people and be my small contribution back to PHPDig.

11-23-2004, 08:17 AM	#5
Ensim Green Mole Join Date: Nov 2004 Posts: 4	Thanks again Vinyl-Junkie. I gave up on the command line version, I've had problems with command line php before though, it doesn't appear to be the same exact php that runs in apache on our server. For those interested, here is the command line I came up with that works, it uses wget to call the php via apache: /usr/bin/wget --timeout 3600 --http-user your_htaccess_username --http-passwd your_htaccess_password "http://www.mysite.org/phpdig/admin/update.php?path=&site_id=1&exp=1" 1>/dev/null 2>/dev/null (where it is all one line with no breaks) Here's the explanation of the command: /usr/bin/wget Replace this with what ever path there is to your wget, find out using "which wget" at the SSH/telnet prompt. --timeout 3600 this is the timeout in seconds. I removed the sleep(5) in the spider.php, if you keep it there increase this timeout a lot, my update.php call takes 20 seconds so an hour should be plenty for me. I also removed all the "set_timeout_limit" calls from PHPDig since my PHP safe mode gave errors that resulted in "update form" not reindexing the site. --http-user & --http_passwd I turned off the PHPDig admin user/pass by updating this in config.php: define('PHPDIG_ADM_AUTH','0'); Then I password protected the admin directory using apache .htaccess files, and included their username and password as these parameters. "http://www.mysite.org/phpdig/admin/update.php?path=&site_id=1&exp=1" This link should work in your browser (after logging in) and force a site reindex. I got this link by selecting my site in the admin and clicking "update form", then copying the link that is there on the "root" green checkbox. I had to include the quotes so wget understood the full link. 1>/dev/null 2>/dev/null This is optional. Without it, each time this command runs wget will save another copy of the page loaded which you may want as a record of the indexing. By including this, no files are created by wget. Then you can add this to your cronjob so it can reindex automatically. Hopefully this will help people and be my small contribution back to PHPDig.