Hi I have php dig set up on a web site and I'm having trouble getting round a problem.
PHP DIG indexes pages when spidering, these pages are roots of folders, they appear in the spider directory as '-' entries, these are then listed when a user searches, when they are clicked they just open folder pages on my web site such as:
www.domain.com/html/one/
thats a bit confusing so try this description:
if you have a page called:
www.domain.com/html/one/one.html
the spider indexes this link but also indexes the root folder of that html page, so you get both of these results:
www.domain.com/html/one/one.html
www.domain.com/html/one/
I don't want these folders indexed, i just want the html pages indexed,
SO, does anyone know how to get round this, your help will be much appreciated.