>> As the pages are creating different $md5 variables, they are not seen as duplicates.
Hi. If you are using the following:
PHP Code:
define('PHPDIG_SESSID_REMOVE',true); // remove SIDS from indexed URLS
define('PHPDIG_SESSID_VAR','MoodleSession'); // name of the SID variable
and the same page shows up multiple times in a crawl, then refresh the page in your browser. Does something on the page change, like a banner?
If so, then try using the PHPDIG_EXCLUDE_COMMENT and PHPDIG_INCLUDE_COMMENT values to surround the part of the page that is changing upon refresh.