OK - here is what the 3rd level support at my host says:
The script was being killed because it was using up too much CPU time. The
maximum amount of CPU time a process can use is 20%. This script was
regularly using 80-90% of the cpu cycles on this machine, which is
unacceptable in a shared hosting environment.
One alternative may be to run the script with a different niceness value.
This can be done using:
nice --adjust=19 /usr/bin/php4 -f spider.php
http://www.globalwaterintel.com/list.php
Adjust can be any value between 0 (normal priority) or 19 (as nice as
possible).
If you just place lots of sleeps in the code, then what may happen is that
the program uses no CPU time, then uses a large amount for a short burst.
If the process monitor happens to see it during a short burst of high
activity, then it may still kill it.
The thing that I don't understand is, why does the broswer version run OK. Surely it would use more resource than running it from the shell as it is having to output to HTML - which I assume is buffered.