Excluding only one link
 

I have been walking around with this problem too long. I hope someone can help.

I have quite a big site, when I spider it, I get about 8000 pages. But most of them are duplicate, about 6500 of them.

Those are the pages to compose a message to a forum poster. Like:
.../forum/messagecompose.asp?senduser=pluimenest&topic=1227&recordnum=20

I tried taking out the variable senduser. (the others, topic and recordnum I cannot take out, cause they are used on other pages as well)

I also tried using phpdigInclude and phpdigExclude to not get that page indexed. The page is out of the searchresults, but they still get spidered. 6500 times a page that is spiderred but not indexed, still takes 9 hours. ( I know I can set the interval time lower, but thats not a sollution)

I do want to have something, so the file messagecompose.asp won't be spiderred at all. Easiest would be if there is a way to have an exclude/include tag that won't follow links between the tags. But not ruining the orginal exclude/include tags.

This way I only will have to update one page, setting the tags, and I lose 6500 indexed pages, and have a gain of 9 hours. Anyone can help me with this?

Thanx

Try sticking messagecompose.asp in a robots.txt file.

This is in my robots.txt :

User-agent: PhpDig
Disallow: /pcs/philboard_reply.asp
Disallow: /pcs/messagecompose.asp
Disallow: /pcs/nologin.asp

as you can see there are more sites I don't want to index.
But it's not working, I started spiderring again, and still all those pages are spiderred.

So the bot still spiders them all.
Isn't there a way to make phpinclude/phpexclude to not follow links?

Anyone any idea how to stop those links to be spidered?

This is just a guess at a solution, but go into your admin panel and delete one of those pages from your index that you don't want there. Now, with your robots.txt in place as discussed above, try indexing just that one page and see what phpdig does with it.

I'm thinking that perhaps phpdig isn't clearing out old url's that you don't want in the index, but with them gone from your database and rules in place to exclude them from future indexing, maybe phpdig won't try to add them back in.

Let us know if that worked or not.

I feel so stupid.

It works now.
Only one "s" was the problem, I had my robot text file called: robot.txt
After changing it to robots.txt it works great.

So stupid of me.
Thanx for all the help.