PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Mod Requests

Reply
 
Thread Tools
Old 03-29-2004, 09:37 PM   #1
jerrywin5
Orange Mole
 
Join Date: Mar 2004
Posts: 48
Admin approval for spider to index external URLs

Letting the spider run wild does a niche search engine more harm than good. As it indexes sites that are not relevant, it uses server resources unnecessarily and causes search results to slow down due to additional load.

A modification I'd really like to see is to have the spider place all external links it finds while indexing a site into a table for later admin approval to index. Upon approval, these urls could be moved to a text file for indexing.
jerrywin5 is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
I: PHPDIG can not index 2+ URLs.. ? PL_90 Script Installation 0 10-22-2007 07:51 AM
Restart spider and index urls in temptable jerrywin5 Troubleshooting 1 04-06-2005 01:18 PM
Spider External links to a depth of 1 (1.8.3) kenazo How-to Forum 0 10-20-2004 06:28 AM
phpdig seems to guess some urls and spider it manute Troubleshooting 7 04-29-2004 01:49 AM
viewing admin/index.php... ronyotz Troubleshooting 3 03-24-2004 04:33 AM


All times are GMT -8. The time now is 04:53 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.