sleep(5) in spider.php?
 

Anybody know why the sleep(5) is in the big loop in spider.php? I took it out and surprisingly saved 5 seconds per file indexed. Besides the obvious consequences of not giving the CPU a rest, is there any good reason to have a sleep?

Hi. If you crawl sites that are not yours, setting a delay is the courteous thing to do so that you do not place too high of a load on the other person's machine.

Why don't make a settings for that? for example "force indexing!" :) And when it's on, indexing is speeded up. Or force every 5 urls and sleep after that for a 5 secs, then again... and so on...