PhpDig.net - View Single Post - Spider indexes cgi pages but not its links!?

WebSpider · 02-08-2005, 02:18 PM

But then, why search results do not provide any cgi.domain.com result BUT will index it as keywords? how do i remove those false keywords and rerun the spider so it will pick cgi. as a url and not as a keyword?

02-08-2005, 02:18 PM	#3
WebSpider Green Mole Join Date: Feb 2005 Posts: 16	But then, why search results do not provide any cgi.domain.com result BUT will index it as keywords? how do i remove those false keywords and rerun the spider so it will pick cgi. as a url and not as a keyword?