|
Once the spider makes an initial check, it could set a field in the sites table and then include or not include the desciption and/or keywords each time it indexes the site based on the value of the field. It could also check the setting avery so many indexes of the site after x number of indexes. Ignoring the description and keywords of all sites means not taking advantage of their value. Not ignoring the description and/or keywords when they are duplicated throughout the site means users will get bogus search results. So, there needs to be some way to handle this issue.
|