View Single Post
Old 01-11-2004, 02:49 PM   #2
siliconkibou
Green Mole
 
Join Date: Dec 2003
Posts: 11
Thanks for the tutorial...

Questions:

Will this update or simply fresh-spider every site in the text file list?

In connection, will this method use stored usernames and passwords for password protected sites?

What sort of server load(on average) will running spiders on a whole list of sites create?

Lastly, wouldn't it be simpler to setup a cron job to run a "spiderupdate.php" or equivilant?

spiderupdate.php could pull all the URLs out of the database and spider them according to the config settings.

Beats manually entering several hundred URL's(although one could probably export a text file with the URL's from the database table, as well).

Thanks,

-Paul
siliconkibou is offline   Reply With Quote