index only .shtml

Nosmada · 12-23-2003, 09:53 AM

How do you index only .shtml and avoid all other file types such as .txt or .html or .htm...

Charter · 12-23-2003, 10:08 AM

Hi. Try adding the unwanted file extensions to the FORBIDDEN_EXTENSIONS in the config file.

Nosmada · 12-23-2003, 10:27 AM

Thanks Charter. Almost there! I have FrontPage extensions installed and have many folders in many directories that start with an underscore (_folder). What syntax do I use in the robots.txt to exclude all folders that start with an underscore?

Would I use something like this?

Disallow: /_

or maybe

Disallow: /_*

or something else???

Charter · 12-23-2003, 03:36 PM

Hi. If you've already crawled directories that start with an underscore, you can delete them from the admin panel by clicking the site, the update button, and then the appropriate red X symbol.

To exclude all folders that start with an underscore, the robots.txt file can be like so (wildcards are nonstandard):

User-agent: PhpDig
Disallow: /_

12-23-2003, 09:53 AM	#1
Nosmada Orange Mole Join Date: Dec 2003 Posts: 32	index only .shtml How do you index only .shtml and avoid all other file types such as .txt or .html or .htm... __________________ Nosmada

12-23-2003, 10:08 AM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Try adding the unwanted file extensions to the FORBIDDEN_EXTENSIONS in the config file. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

12-23-2003, 10:27 AM	#3
Nosmada Orange Mole Join Date: Dec 2003 Posts: 32	Thanks Charter. Almost there! I have FrontPage extensions installed and have many folders in many directories that start with an underscore (_folder). What syntax do I use in the robots.txt to exclude all folders that start with an underscore? Would I use something like this? Disallow: /_ or maybe Disallow: /_* or something else??? __________________ Nosmada

12-23-2003, 03:36 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. If you've already crawled directories that start with an underscore, you can delete them from the admin panel by clicking the site, the update button, and then the appropriate red X symbol. To exclude all folders that start with an underscore, the robots.txt file can be like so (wildcards are nonstandard): User-agent: PhpDig Disallow: /_ __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
.shtml as a template?	sparky	How-to Forum	6	09-28-2004 06:13 PM
Adding index.shtml	urb	How-to Forum	1	02-23-2004 09:24 AM
.shtml pages and including borders	Nosmada	How-to Forum	7	01-08-2004 12:04 AM