PhpDig.net - View Single Post - Admin approval for spider to index external URLs

jerrywin5 · 03-29-2004, 09:37 PM

Letting the spider run wild does a niche search engine more harm than good. As it indexes sites that are not relevant, it uses server resources unnecessarily and causes search results to slow down due to additional load.

A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing.

03-29-2004, 09:37 PM	#1
jerrywin5 Orange Mole Join Date: Mar 2004 Posts: 48	Admin approval for spider to index external URLs Letting the spider run wild does a niche search engine more harm than good. As it indexes sites that are not relevant, it uses server resources unnecessarily and causes search results to slow down due to additional load. A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing.