Beans
08-14-2005, 01:21 PM
My goal is a topic specific search engine.. many web resources for this topic are scattered across the web on various servers.. some deticated, some not.
For instance, let's say i'm running a search engine just for information about golf balls.
A website like "www.justgolfballs.com" might have information contained in the directory /information and sales stuff in a directory called /sales
I don't want to include the /sales directory in my indexing, because if somebody searches for "good balls" I want them to see information, not a sales pitch.
Now a website like "www.justgolfballinformation.com" might contain nothing but golf ball information.. so this is a site i'd like to search all of.
So how do I limit my indexed pages to a specific directory for certian domains?
For instance, let's say i'm running a search engine just for information about golf balls.
A website like "www.justgolfballs.com" might have information contained in the directory /information and sales stuff in a directory called /sales
I don't want to include the /sales directory in my indexing, because if somebody searches for "good balls" I want them to see information, not a sales pitch.
Now a website like "www.justgolfballinformation.com" might contain nothing but golf ball information.. so this is a site i'd like to search all of.
So how do I limit my indexed pages to a specific directory for certian domains?