PhpDig.net - View Single Post

Charter · 08-07-2004, 11:07 AM

Hi. Just stick a robots.txt file in web root. PhpDig will look for the file at http://www.domain.com/robots.txt but if you already crawled directories that you want to exclude, then you can delete and/or exclude them from the PhpDig admin panel. With version 1.8.2+ you can also limit the index to a specific directory. If you have PhpDig crawl http://www.domain.com/dir/ (with ending slash) then it should only crawl content that resides in that directory.

08-07-2004, 11:07 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. Just stick a robots.txt file in web root. PhpDig will look for the file at http://www.domain.com/robots.txt but if you already crawled directories that you want to exclude, then you can delete and/or exclude them from the PhpDig admin panel. With version 1.8.2+ you can also limit the index to a specific directory. If you have PhpDig crawl http://www.domain.com/dir/ (with ending slash) then it should only crawl content that resides in that directory. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.