PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)
-   -   How to exclude subdirectories while indexing a site (http://www.phpdig.net/forum/showthread.php?t=1492)

philbihr 11-03-2004 01:54 AM

How to exclude subdirectories while indexing a site
 
Good morning,
Is it possible, when indexing a site, to exclude all or some subdirectories ?
Thanks :)

philbihr 11-08-2004 04:23 AM

I have found this in documentation

=================================================
3.4. Other features
-------------------------


PhpDig Tries to read a robots.txt file at the server root.
It searches meta robots tags too.
==================================================

Please how does this feature function ? How are the robot.txt record's. Where is this file ?

philbihr 11-10-2004 02:48 AM

I have found what I was seeking for.
The solution is to create a "robots.txt" exclusion file at the root of the site.
Informations about robots exclusion at :

http://www.robotstxt.org/wc/norobots.html


All times are GMT -8. The time now is 02:14 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.