PhpDig.net - View Single Post

Charter · 01-07-2004, 12:51 PM

Hi. You can set a robots.txt file with the following content:

Code:

# http://www.mywebsite.com/robots.txt

User-agent: PhpDig
Disallow: /Dep1
Disallow: /Dep3
Disallow: /Dep4

# and so forth

If you already crawled these directories, you can exlude and delete them from the control panel.

01-07-2004, 12:51 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. You can set a robots.txt file with the following content: Code: # http://www.mywebsite.com/robots.txt User-agent: PhpDig Disallow: /Dep1 Disallow: /Dep3 Disallow: /Dep4 # and so forth If you already crawled these directories, you can exlude and delete them from the control panel. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.