|
Hi misterbear -
We definitely need a way to run multiple spiders, but I think running separate versions on separate databases is sort of like hiring two slow typists and paying them double instead of just hiring a fast one and paying him the regular wage. What I mean to say is, it doesn't really solve the problem. I think the solution will involve one database, and multiple running spider processes that are smart enough to know which sites are being spidered already. In this way, we can limit redundant data and tables, and simplify the process for people that are allowed only one database with their host (for example, freeservers). But thanks for trying to work out solutions!
|