![]() |
How to exclude subdirectories while indexing a site
Good morning,
Is it possible, when indexing a site, to exclude all or some subdirectories ? Thanks :) |
I have found this in documentation
================================================= 3.4. Other features ------------------------- PhpDig Tries to read a robots.txt file at the server root. It searches meta robots tags too. ================================================== Please how does this feature function ? How are the robot.txt record's. Where is this file ? |
I have found what I was seeking for.
The solution is to create a "robots.txt" exclusion file at the root of the site. Informations about robots exclusion at : http://www.robotstxt.org/wc/norobots.html |
All times are GMT -8. The time now is 02:14 AM. |
Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.