arena75
10-04-2004, 06:20 AM
I have been walking around with this problem too long. I hope someone can help.
I have quite a big site, when I spider it, I get about 8000 pages. But most of them are duplicate, about 6500 of them.
Those are the pages to compose a message to a forum poster. Like:
.../forum/messagecompose.asp?senduser=pluimenest&topic=1227&recordnum=20
I tried taking out the variable senduser. (the others, topic and recordnum I cannot take out, cause they are used on other pages as well)
I also tried using phpdigInclude and phpdigExclude to not get that page indexed. The page is out of the searchresults, but they still get spidered. 6500 times a page that is spiderred but not indexed, still takes 9 hours. ( I know I can set the interval time lower, but thats not a sollution)
I do want to have something, so the file messagecompose.asp won't be spiderred at all. Easiest would be if there is a way to have an exclude/include tag that won't follow links between the tags. But not ruining the orginal exclude/include tags.
This way I only will have to update one page, setting the tags, and I lose 6500 indexed pages, and have a gain of 9 hours. Anyone can help me with this?
Thanx
I have quite a big site, when I spider it, I get about 8000 pages. But most of them are duplicate, about 6500 of them.
Those are the pages to compose a message to a forum poster. Like:
.../forum/messagecompose.asp?senduser=pluimenest&topic=1227&recordnum=20
I tried taking out the variable senduser. (the others, topic and recordnum I cannot take out, cause they are used on other pages as well)
I also tried using phpdigInclude and phpdigExclude to not get that page indexed. The page is out of the searchresults, but they still get spidered. 6500 times a page that is spiderred but not indexed, still takes 9 hours. ( I know I can set the interval time lower, but thats not a sollution)
I do want to have something, so the file messagecompose.asp won't be spiderred at all. Easiest would be if there is a way to have an exclude/include tag that won't follow links between the tags. But not ruining the orginal exclude/include tags.
This way I only will have to update one page, setting the tags, and I lose 6500 indexed pages, and have a gain of 9 hours. Anyone can help me with this?
Thanx