Quote:
|
Originally Posted by Slider
www.domain.com/links/file.ext
Note the part of the url in bold is the part I need banned.
Not a file or an extension. A DIRECTORY to a file. Also called the path
I need the directory called LINKS as a banned directory.
If it gets to that directory it doesn't follow it to spider any further.
I honestly do appreciate your help. A answer of any kind is better than no answer at all.
|
Just use your robots.txt file to do that, like so:
Code:
User-agent: Phpdig
Disallow: /links/