Samar
02-09-2006, 09:55 AM
Hi,
I have a robot.txt file at the root of my website that I use to exclude some path from being indexed.
Let's say I want to exclude the folder /en/ and all is content from being indexed.
The code im using in the robot.txt
User-agent: *
Disallow: /en/
My problem is that PHPdig is still indexing all the files under /en/
Anything I could do?
I have a robot.txt file at the root of my website that I use to exclude some path from being indexed.
Let's say I want to exclude the folder /en/ and all is content from being indexed.
The code im using in the robot.txt
User-agent: *
Disallow: /en/
My problem is that PHPdig is still indexing all the files under /en/
Anything I could do?