PhpDig.net - Not index certain links?

PhpDig.net (http://www.phpdig.net/forum/index.php)

- How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)

- - Not index certain links? (http://www.phpdig.net/forum/showthread.php?t=2204)

Not index certain links?

Hello,

how can I configure that certain links are not indexed and contained in the search?

Example: I have a link on one site to mydomain.com/mysite.html, I don't want the spider to follow this link and index it to the search database.

Thanks for help!

Add mysite.html to your robots.txt file or click the 'no way' sign for the page from the admin panel. Alternatively, check out the following code in the config file and see this thread.

Code:

// in PhpDig 1.8.8 RC1



// regexp for forbidden links - some links may return text/html mime-type but should not be indexed so forbid them !!!

// you can also expand the FORBIDDEN value by writing a regex to forbid certain links containing 'word' from being indexed

define('FORBIDDEN','\.(js|inc|rm|ico|cab|swf|css|gz|z|tar|zip|tgz|msi|arj|zoo|rar|r[0-9]+|exe|bin|pkg|rpm|deb|bz2)$');



// in PhpDig 1.8.7



// regexp forbidden extensions - return sometimes text/html mime-type !!!

define('FORBIDDEN_EXTENSIONS','\.(rm|ico|cab|swf|css|gz|z|tar|zip|tgz|msi|arj|zoo|rar|r[0-9]+|exe|bin|pkg|rpm|deb|bz2)$');

Sorry ;) i found out by myself how it works. You only have to create a robots.txt on the root fo the server with the following content:

User-agent: PhpDig
Disallow: /folder/filetoexclude.extension