PhpDig.net - View Single Post - How to exclude subdirectories while indexing a site

philbihr · 11-10-2004, 02:48 AM

I have found what I was seeking for.
The solution is to create a "robots.txt" exclusion file at the root of the site.
Informations about robots exclusion at :

http://www.robotstxt.org/wc/norobots.html

11-10-2004, 02:48 AM	#3
philbihr Green Mole Join Date: Oct 2004 Location: Senlis (60) France Posts: 18	I have found what I was seeking for. The solution is to create a "robots.txt" exclusion file at the root of the site. Informations about robots exclusion at : http://www.robotstxt.org/wc/norobots.html