View Full Version : sleep(5) in spider.php?

02-18-2004, 06:47 PM
Anybody know why the sleep(5) is in the big loop in spider.php? I took it out and surprisingly saved 5 seconds per file indexed. Besides the obvious consequences of not giving the CPU a rest, is there any good reason to have a sleep?

02-19-2004, 06:40 AM
Hi. If you crawl sites that are not yours, setting a delay is the courteous thing to do so that you do not place too high of a load on the other person's machine.

03-12-2004, 02:31 PM
Why don't make a settings for that? for example "force indexing!" :) And when it's on, indexing is speeded up. Or force every 5 urls and sleep after that for a 5 secs, then again... and so on...