PhpDig.net

Go Back   PhpDig.net > General Forums > The Mole Hole

Reply
 
Thread Tools
Old 01-11-2005, 02:19 AM   #1
frak
Green Mole
 
Join Date: Jan 2005
Posts: 2
hello / directories / phpdig & others

Hi,
I've been a fan of phpDig for a long time now. I have had it installed to test on a small scale about a year ago.

I've got the the point where I would like to do a large search engine. I'm a bit concerned at a few poeple talking about effective size limits of 35k-70k indexed pages (slow search performance)

I would be looking at an index larger than that. Is this something that php-dig can index quickly enough? (ie non-instant percieved results would not be good enough)

Or am I beter off going with something like mnoGoSearch? (depending on the answer I will be setting up an indexer here on my dev server this week to give it a good test thrashing)

I am looking at doing something interesting with whatever I end up going with - I'll post details on exactly what later...

Also - can somebody recommend Directory software (ala ODP)?

Cheers,
Mathew
frak is offline   Reply With Quote
Old 01-11-2005, 04:59 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
As I have not tried mnoGoSearch, I cannot give you any comparison information. If you wish to make a large scale search engine, then you should consider that you'll probably need a cluster of servers to process requests. Also, you'll probably want to run precompiled code rather than parse code on each run, utilize a caching system, send compressed output, etcetera. Having a server and a script is not enough to go large scale. As for a script directory, there used to be something called "PHP Script Index" but I'm not sure if it's still available.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 01-12-2005, 10:38 PM   #3
frak
Green Mole
 
Join Date: Jan 2005
Posts: 2
thanks - one more question

Charter,
You have confirmed what I suspected from my own research. Pity.

It would seem that the best performance - a few mill pages indexed at <2 sec - is acheived with DataPark followed by mnogo.

I do have a question though - it seems like a alot of the "grunt" work for SEs is done by script/bins outside of the DB, instead of the database server? I had thought that the DB would do the hard work.

Why is that?

Cheers,
Mathew
frak is offline   Reply With Quote
Old 01-12-2005, 11:18 PM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Maybe this thread can answer your DB question, at least WRT PhpDig.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
how much disk&db space need for first phpdig install 01php Script Installation 1 03-21-2007 07:09 PM
Url part after &amp; is ignored on spider (1.8.9 RC1 and earliers) obottek Bug Tracker 1 08-24-2006 04:52 AM
phpdig & https desfaitl How-to Forum 1 09-11-2004 06:41 AM
PhpDig & UTF-8 naskida How-to Forum 1 12-27-2003 04:44 PM
PhpDig & PHP-Nuke j-ortega How-to Forum 4 11-16-2003 11:28 PM


All times are GMT -8. The time now is 03:32 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.