Go Back > PhpDig Forums > Troubleshooting

Thread Tools
Old 04-26-2004, 04:09 PM   #1
Orange Mole
Join Date: Jan 2004
Posts: 30
Set time limit on spider.php or number of pages


I use a commercial server which has limits to the amount of resources I can use with spider.php. They've warned me before about the script taking up too many resources and are threatening to shut me down. I usually give spider.php a list of 30 sites (level0 spidering depth) to spider at a time as a cron job

Can I limit spider.php to run a maximim of 10 minutes even if it hasn't completed a cron job on a list of url's, to help prevent my site from being shut down by an angry administrator?


Paul L
paullind is offline   Reply With Quote
Old 05-01-2004, 07:25 AM   #2
Head Mole
Charter's Avatar
Join Date: May 2003
Posts: 2,539
Hi. If your host allows, set a cron job like so:
10 * * * * ps -ef | grep 'php -f spider.php' | awk '{print $2}' | xargs kill -9

10 * * * *               : complete this line to be 10 minutes after spider start
ps -ef                   : list out the processes
grep 'php -f spider.php' : find the spider process
awk '{print $2}'         : get PID from column two
xargs kill -9            : kill the spider process
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
set time limit Macfplus Script Installation 0 11-17-2006 08:03 AM
Number of pages indexed claudiomet How-to Forum 0 08-30-2004 02:26 PM
I get a warning - Cannot set time limit!! macin1818 Troubleshooting 1 04-20-2004 10:51 AM
Spider & time limit onlytrue How-to Forum 1 04-16-2004 05:03 AM
Limit number of spidered pages Not Logged In How-to Forum 5 12-16-2003 03:03 PM

All times are GMT -8. The time now is 02:48 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.