Yes, there are some differences in my config file between 1.8.0 and 1.8.1. I'm not sure whether they'd make that much difference though. Here are the ones that have anything to do with spidering:
SPIDER_MAX_LIMIT 20 (1.8.0) vs. 10 (1.8.1)
SPIDER_DEFAULT_LIMIT 3 (1.8.0) vs. 5 (1.8.1)
Everything else is the same in both versions.
I'll copy the 1.8.1 config file to my server, re-spider and let you know what happens.
|