|
Thanks, but if I use a robots.txt file to exclude certain directories, won't that prevent those dirs from being indexed by public search engines too (e.g. Google?).
I'm only trying to fine-tune our search - for example, I'd like to exclude our forums from all searches, and I'd like to remove our Developers area from all non-tech releated searches.
Should I be excluding different directories and running separate indexes, or should I be running one large index and (if possible?) excluding parts of the site at search-time?
-Antun
|