PhpDig.net - View Single Post

WebSpider · 02-09-2005, 02:18 PM

- Used both text and combo box
- No robots.txt present
- No revisits on the code
- Thats the only thing i should try now. However, does it make sense to index both www.domain.com and domain.com when they're 99% of the times the same thing? shouldn't this be implemented (even as a switch?) on the code of the digger?

02-09-2005, 02:18 PM	#9
WebSpider Green Mole Join Date: Feb 2005 Posts: 16	- Used both text and combo box - No robots.txt present - No revisits on the code - Thats the only thing i should try now. However, does it make sense to index both www.domain.com and domain.com when they're 99% of the times the same thing? shouldn't this be implemented (even as a switch?) on the code of the digger?