View Single Post
Old 03-14-2004, 12:03 AM   #1
Konstantine
Green Mole
 
Join Date: Mar 2004
Location: Russia
Posts: 21
Logical bug and stopping spidering

When browser stops by user or timeout at the time of retrieving of urls some urls won't be save cause page was marked as indexed and won't be search again for links after refreshing. Am I right? So it's not a problem when you have no limit of time and start spidering by command line, but for browser or timeout...
Konstantine is offline   Reply With Quote