>> I also started a spider with the exec() command, which ran for a while and then stopped with links still in the temporary table and without unlocking. Most likely this is because I didn't set links per for this spider?
Hi. Hmm, not sure on this one. In spider.php is the following:
PHP Code:
if (!isset($linksper) or (int)$linksper > LINKS_MAX_LIMIT) {
if ($run_mode != 'cgi') {
$linksper = RELINKS_LIMIT;
}
else {
$linksper = LINKS_MAX_LIMIT;
}
}
In 1.8.1 the links per is either set via the browser interface or by values in the config file. One of the new tables has links per for each site, but utilizing this table didn't get done for version 1.8.1, so for now, links per will be the same for all sites crawled via shell.
Anyway, back to your exec issue, I'm not sure why the spider quit. Maybe r****m noise, maybe not. Anything in your error logs?