Thread: Forking when spidering
View Single Post
Old 10-12-2004, 09:40 PM   #12
shamu
Green Mole
 
Join Date: Oct 2004
Posts: 4
So is there no way to fix the code to respect robots.txt?

I'm just wondering.
shamu is offline   Reply With Quote