|
01-11-2005, 03:19 AM | #1 |
Green Mole
Join Date: Jan 2005
Posts: 2
|
hello / directories / phpdig & others
Hi,
I've been a fan of phpDig for a long time now. I have had it installed to test on a small scale about a year ago. I've got the the point where I would like to do a large search engine. I'm a bit concerned at a few poeple talking about effective size limits of 35k-70k indexed pages (slow search performance) I would be looking at an index larger than that. Is this something that php-dig can index quickly enough? (ie non-instant percieved results would not be good enough) Or am I beter off going with something like mnoGoSearch? (depending on the answer I will be setting up an indexer here on my dev server this week to give it a good test thrashing) I am looking at doing something interesting with whatever I end up going with - I'll post details on exactly what later... Also - can somebody recommend Directory software (ala ODP)? Cheers, Mathew |
01-11-2005, 05:59 AM | #2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
As I have not tried mnoGoSearch, I cannot give you any comparison information. If you wish to make a large scale search engine, then you should consider that you'll probably need a cluster of servers to process requests. Also, you'll probably want to run precompiled code rather than parse code on each run, utilize a caching system, send compressed output, etcetera. Having a server and a script is not enough to go large scale. As for a script directory, there used to be something called "PHP Script Index" but I'm not sure if it's still available.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
01-12-2005, 11:38 PM | #3 |
Green Mole
Join Date: Jan 2005
Posts: 2
|
thanks - one more question
Charter,
You have confirmed what I suspected from my own research. Pity. It would seem that the best performance - a few mill pages indexed at <2 sec - is acheived with DataPark followed by mnogo. I do have a question though - it seems like a alot of the "grunt" work for SEs is done by script/bins outside of the DB, instead of the database server? I had thought that the DB would do the hard work. Why is that? Cheers, Mathew |
01-13-2005, 12:18 AM | #4 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Maybe this thread can answer your DB question, at least WRT PhpDig.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
Thread Tools | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
how much disk&db space need for first phpdig install | 01php | Script Installation | 1 | 03-21-2007 08:09 PM |
Url part after & is ignored on spider (1.8.9 RC1 and earliers) | obottek | Bug Tracker | 1 | 08-24-2006 05:52 AM |
phpdig & https | desfaitl | How-to Forum | 1 | 09-11-2004 07:41 AM |
PhpDig & UTF-8 | naskida | How-to Forum | 1 | 12-27-2003 05:44 PM |
PhpDig & PHP-Nuke | j-ortega | How-to Forum | 4 | 11-17-2003 12:28 AM |