Hi!
When I run the spider on a site
www.domain.com that hosts several pages in the form of
www.domain.com/cgi-bin/whatever... or cgi-bin.domain.com/whatever.... I can't find those links on the database nor the search results, but checking the most common keywords gives me as 1st place the cgi-bin.domain.com keyword.
What's the deal?
How do i make it to add the cgi-*.* links to the database for that particular domain?
Also, is there any difference between indexing
http://www.domain.com and
http://domain.com? Will I get duplicate pages onto the db?
Thanks!