PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Mod Requests

Reply
 
Thread Tools
Old 05-22-2004, 05:51 PM   #1
2wheelin
Green Mole
 
Join Date: Apr 2004
Posts: 3
Spidering multiple URL's

I need a search/spider program to index subject specific web sites.

As user RaGe outlined below, phpDig is not able to index multiple URL's as far as I can see. Why not add this ability and make it an effective Web Search Engine?

BTW- phpDig is a GREAT script for a single URL (site search), the best I have found! Add the URL feature and phpDig will be the best of both worlds.

Quote:
Thus far i've seen the spider functions only deal with spidering a particular site and returning only results within the spidered URL. An option that would allow the Admin to ignore the base URL and return only links to external URL's would allow for spidering of a link farm site or links page and harvesting the links back into PhP dig. For example:

I built a cgi engine and have tons of links indexed on it, if i use PhP dig to try to spider the links from the original engine, it returns MY url links instead of ignoring base url and spidering the external links at a depth of 1. Thus it is a URL harvester spider rather than just a site spider.

My cgi engine does this with the greatest of ease, i can spider a particular directory of DMOZ and bring back only the links and their relative URLS. If someone out there (in the Mole Squad) is proficient at both PhP and CGI i'd be willing to make my engine available and perhaps we can cross the spider functions into PhP dig and save some raw coding time for all.

It also features admin features for visitor added URL's that can be directly edited rather than just spidered. At this time i see no way of editing spidered or user submitted urls without doing such at an SQL level which might also be a useful PhPdig function to consider.
2wheelin is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
SEF URL's? raustin Mod Requests 0 07-22-2008 10:39 AM
Multiple Spiders jmitchell How-to Forum 3 12-16-2004 04:43 PM
Calling all persons spidering multiple domains Slider How-to Forum 2 11-06-2004 01:12 AM
QUESTION: How-to Spider Multiple URL's, not just one at a time. 2wheelin How-to Forum 4 06-13-2004 10:42 PM
Using text file for URL's jimigisme How-to Forum 1 09-30-2003 11:53 PM


All times are GMT -8. The time now is 10:59 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.