|
when spidering via text file list of domains ...
can we set it to only take a couple pages at a time from each domain in the list and then move on to the next. Premise being that when a couple pages from each domain are taken to move to the top of the list and take a couple more ?
that would make it so we aren't hammering sites too hard when we spider them
|