|
Ban features
When doing a huge list of sites to index I may add a list and not look through simply because it is so huge of a list.
In the config:
- AutoBan certain Domains. EX: freeservers Geocities and others
- AutoBan universal Directory names: links, banners, affiliates, forums, blog, webrings, Etc.. (not just for a certain site, but for all that are crawled) I'm understanding that this is in Phpdig already, but I have yet to get it working.
- Autoban certain domains by ext: .biz .info .this .that (if you run an english only site this would be very helpful to weed out sites like .ru and others)
This would be a great feature to reduce the size of a bloated Mysq Database.
|