PDA

View Full Version : Problem with links


marsupillami
09-07-2003, 12:49 PM
Hi. Just installed phpDig, because I´m searching for a search engine for my website. First I tried to make it dig since the root, and it found many links. However, I only wanted it to search the following files:

http://www.oc-zone.com/modules.php&name=Reviews
http://www.oc-zone.com/modules.php&name=Articles
http://www.oc-zone.com/modules.php&name=News

How can this be done?

If I try to do it with these URL´s, I get no new links :(

Thank you for any help.

tom5989
09-08-2003, 04:00 PM
You need to create a robot.txt file in your root directory that excludes all but those directories. There is a free program at download.com that will create a robot file for you. Just search for it.

marsupillami
09-08-2003, 04:36 PM
Thank you :)

Charter
09-08-2003, 04:56 PM
Hi. Here are a couple of simple robots.txt examples:

# Disallow PhpDig from dir_to_exclude

User-agent: PhpDig
Disallow: /dir_to_exclude/

# Disallow all robots from all directories
# assuming the robots obey robots.txt

User-agent: *
Disallow: /