jerrywin5
03-29-2004, 09:37 PM
Letting the spider run wild does a niche search engine more harm than good. As it indexes sites that are not relevant, it uses server resources unnecessarily and causes search results to slow down due to additional load.
A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing.
A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing.