PhpDig.net - View Single Post

vinyl-junkie · 12-28-2004, 03:16 AM

Check the robots.txt file for this site. It's possible you don't have permission to spider it.

12-28-2004, 03:16 AM	#7
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Check the robots.txt file for this site. It's possible you don't have permission to spider it.