[ChadK]

When you search on my site for something like "Macaw" you will get a bunch of results. The problem is, many of these results are the same thing with slightly different URLs. You may see:

URL/index.php?showthread=1
this is the snippet with macaw in it

URL/index.php?markthread=1
this is the snippet with macaw in it

URL/index.php?reply=1
this is the snippet with macaw in it

URL/index.php?notification=1
this is the snippet with macaw in it



and so on... they're all essentially the same page it's just showing the links to actions that can be taken on those pages because those links also show the original post in the action. So if you click REPLY, you see the original post... but it's a unique URL so it's added twice.

Any ideas on how to eliminate this problem?
Maybe we could add something that forces a duplicate flag to mark it as a duplicated page?
MessageIDTags=="showtopic,t";
then as the spider works it's magic, it checks for "showtopic" and "t" in the url and if it's found, it checks the value of "showtopic" or "t" and uses it as a "unique index"? So two urls like this:
URL/index.php?showtopic=12?reply=true
and
URL/index.php?showtopic=12?reply=false
would be marked as "duplicate" and only one would be saved.