Go Back > PhpDig Forums > Troubleshooting

Thread Tools
Old 09-07-2003, 11:49 AM   #1
Green Mole
Join Date: Sep 2003
Posts: 3
Problem with links

Hi. Just installed phpDig, because I´m searching for a search engine for my website. First I tried to make it dig since the root, and it found many links. However, I only wanted it to search the following files:

How can this be done?

If I try to do it with these URL´s, I get no new links

Thank you for any help.
marsupillami is offline   Reply With Quote
Old 09-08-2003, 03:00 PM   #2
Green Mole
Join Date: Sep 2003
Location: USA
Posts: 1
You need to create a robot.txt file in your root directory that excludes all but those directories. There is a free program at that will create a robot file for you. Just search for it.
James Woodford
tom5989 is offline   Reply With Quote
Old 09-08-2003, 03:36 PM   #3
Green Mole
Join Date: Sep 2003
Posts: 3
Thank you
marsupillami is offline   Reply With Quote
Old 09-08-2003, 03:56 PM   #4
Head Mole
Charter's Avatar
Join Date: May 2003
Posts: 2,539
Hi. Here are a couple of simple robots.txt examples:

# Disallow PhpDig from dir_to_exclude

User-agent: PhpDig
Disallow: /dir_to_exclude/

# Disallow all robots from all directories
# assuming the robots obey robots.txt

User-agent: *
Disallow: /
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Spider problem, Search mb_ereg_replace problem. (Fixed?!) cpeter Troubleshooting 0 02-24-2006 01:56 PM
the same problem - not index all links redlock Troubleshooting 0 12-28-2004 06:36 AM
Problem Solved: No links found calvarym Troubleshooting 0 11-10-2004 02:24 PM
Problem with indexing "links found : 0" IAMHHawaii Troubleshooting 1 09-20-2004 12:06 PM
problem indexing dynamic links. orbitalz Troubleshooting 3 04-30-2004 07:47 PM

All times are GMT -8. The time now is 01:17 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.