|
12-05-2003, 01:40 AM | #1 |
Orange Mole
Join Date: Nov 2003
Posts: 69
|
robots.txt
With the following robots.txt, no indexing, I always get: links found: 0, ... Was recently indexed:
User-agent: phpdig Disallow: User-agent: * Disallow: / After removing this robots.txt, all goes fine. My intention was to allow PhpDig to index, but tell the others to go away. Did I get the syntax wrong?
__________________
René Haentjens, Ghent University |
12-05-2003, 03:28 AM | #2 |
Green Mole
Join Date: Dec 2003
Location: Lyon, France
Posts: 17
|
|
12-05-2003, 05:01 AM | #3 |
Orange Mole
Join Date: Nov 2003
Posts: 69
|
I've taken this example from the quoted source, fr. Anonymus. In my opinion it shows that it should be possible (sorry for the lost alignment):
# /robots.txt for http://www.fict.org/ # comments to webmaster@fict.org User-agent: unhipbot Disallow: / User-agent: webcrawler User-agent: excite Disallow: User-agent: * Disallow: /org/plans.html Allow: /org/ Allow: /serv Allow: /~mak Disallow: / The following matrix shows which robots are allowed to access URLs: unhipbot webcrawler-excite other http://www.fict.org/ No Yes No http://www.fict.org/index.html No Yes No http://www.fict.org/robots.txt Yes Yes Yes http://www.fict.org/server.html No Yes Yes http://www.fict.org/services/fast.html No Yes Yes http://www.fict.org/services/slow.html No Yes Yes http://www.fict.org/orgo.gif No Yes No http://www.fict.org/org/about.html No Yes Yes http://www.fict.org/org/plans.html No Yes No http://www.fict.org/%7Ejim/jim.html No Yes No http://www.fict.org/%7Emak/mak.html No Yes Yes
__________________
René Haentjens, Ghent University |
12-05-2003, 02:40 PM | #4 | |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Hi. I haven't tested the below code, but it should get around the following case:
Quote:
In robot_functions.php find the phpdigReadRobotsTxt function and in this function find: PHP Code:
PHP Code:
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
|
Thread Tools | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
robots.txt seems to be ignored :? | galacticvoyager | Bug Tracker | 1 | 11-12-2005 12:52 PM |
robots.txt and URL | djavet | How-to Forum | 4 | 01-11-2005 03:19 AM |
robots.txt comments | edkay | Mod Submissions | 2 | 03-12-2004 12:41 PM |
robots.txt versus robotsxx.txt | Charter | IPs, SEs, & UAs | 0 | 03-11-2004 06:00 PM |
robots.txt ignored | roy | Troubleshooting | 3 | 02-20-2004 08:02 PM |