Problem with links

[marsupillami]

Hi. Just installed phpDig, because I´m searching for a search engine for my website. First I tried to make it dig since the root, and it found many links. However, I only wanted it to search the following files:

http://www.oc-zone.com/modules.php&name=Reviews
http://www.oc-zone.com/modules.php&name=Articles
http://www.oc-zone.com/modules.php&name=News

How can this be done?

If I try to do it with these URL´s, I get no new links :(

Thank you for any help.

[tom5989]

You need to create a robot.txt file in your root directory that excludes all but those directories. There is a free program at download.com that will create a robot file for you. Just search for it.

[marsupillami]

Thank you :)

[Charter]

Hi. Here are a couple of simple robots.txt examples:

# Disallow PhpDig from dir_to_exclude

User-agent: PhpDig
Disallow: /dir_to_exclude/

# Disallow all robots from all directories
# assuming the robots obey robots.txt

User-agent: *
Disallow: /