|
I have found this in documentation
=================================================
3.4. Other features
-------------------------
PhpDig Tries to read a robots.txt file at the server root.
It searches meta robots tags too.
==================================================
Please how does this feature function ? How are the robot.txt record's. Where is this file ?
|