|
The muliple spiders seemed to work when I ran two spiders from the web interface at the same time. Both finished correctly - nice! I also started a spider with the exec() command, which ran for a while and then stopped with links still in the temporary table and without unlocking. Most likely this is because I didn't set links per for this spider? This is the command I used:
exec("/usr/bin/php -f /home/search/admin/spider.php $site 2>&1 > /dev/null &");
As for the directory spidering issue, I think I might play around with the code to get it to do what I want, unless you plan on adding this feature to a future version. Thanks for the help.
|