WebSpider
02-07-2005, 07:01 AM
Hi!
When I run the spider on a site www.domain.com that hosts several pages in the form of www.domain.com/cgi-bin/whatever... or cgi-bin.domain.com/whatever.... I can't find those links on the database nor the search results, but checking the most common keywords gives me as 1st place the cgi-bin.domain.com keyword.
What's the deal?
How do i make it to add the cgi-*.* links to the database for that particular domain?
Also, is there any difference between indexing http://www.domain.com and http://domain.com? Will I get duplicate pages onto the db?
Thanks!
When I run the spider on a site www.domain.com that hosts several pages in the form of www.domain.com/cgi-bin/whatever... or cgi-bin.domain.com/whatever.... I can't find those links on the database nor the search results, but checking the most common keywords gives me as 1st place the cgi-bin.domain.com keyword.
What's the deal?
How do i make it to add the cgi-*.* links to the database for that particular domain?
Also, is there any difference between indexing http://www.domain.com and http://domain.com? Will I get duplicate pages onto the db?
Thanks!