PhpDig.net - View Single Post

Samar · 02-09-2006, 09:55 AM

Hi,

I have a robot.txt file at the root of my website that I use to exclude some path from being indexed.

Let's say I want to exclude the folder /en/ and all is content from being indexed.

The code im using in the robot.txt

User-agent: *
Disallow: /en/

My problem is that PHPdig is still indexing all the files under /en/

Anything I could do?

02-09-2006, 09:55 AM	#1
Samar Green Mole Join Date: Feb 2006 Posts: 3	Problem with robot.txt file Hi, I have a robot.txt file at the root of my website that I use to exclude some path from being indexed. Let's say I want to exclude the folder /en/ and all is content from being indexed. The code im using in the robot.txt User-agent: * Disallow: /en/ My problem is that PHPdig is still indexing all the files under /en/ Anything I could do?