PhpDig.net - View Single Post - How to limit indexing to a specific directory?

Beans · 08-14-2005, 01:21 PM

My goal is a topic specific search engine.. many web resources for this topic are scattered across the web on various servers.. some deticated, some not.

For instance, let's say i'm running a search engine just for information about golf balls.

A website like "www.justgolfballs.com" might have information contained in the directory /information and sales stuff in a directory called /sales

I don't want to include the /sales directory in my indexing, because if somebody searches for "good balls" I want them to see information, not a sales pitch.

Now a website like "www.justgolfballinformation.com" might contain nothing but golf ball information.. so this is a site i'd like to search all of.

So how do I limit my indexed pages to a specific directory for certian domains?

08-14-2005, 01:21 PM	#1
Beans Green Mole Join Date: Jul 2005 Posts: 3	How to limit indexing to a specific directory? My goal is a topic specific search engine.. many web resources for this topic are scattered across the web on various servers.. some deticated, some not. For instance, let's say i'm running a search engine just for information about golf balls. A website like "www.justgolfballs.com" might have information contained in the directory /information and sales stuff in a directory called /sales I don't want to include the /sales directory in my indexing, because if somebody searches for "good balls" I want them to see information, not a sales pitch. Now a website like "www.justgolfballinformation.com" might contain nothing but golf ball information.. so this is a site i'd like to search all of. So how do I limit my indexed pages to a specific directory for certian domains?