Thread: robots.txt
View Single Post
Old 12-05-2003, 01:40 AM   #1
renehaentjens
Orange Mole
 
Join Date: Nov 2003
Posts: 69
robots.txt

With the following robots.txt, no indexing, I always get: links found: 0, ... Was recently indexed:

User-agent: phpdig
Disallow:

User-agent: *
Disallow: /

After removing this robots.txt, all goes fine.

My intention was to allow PhpDig to index, but tell the others to go away. Did I get the syntax wrong?
__________________
René Haentjens, Ghent University
renehaentjens is offline   Reply With Quote