View Single Post
Old 07-12-2006, 01:55 PM   #2
Dave A
Purple Mole
 
Dave A's Avatar
 
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
Phpdig is able to search multiple domains, I have used it so far to do this with around twenty thousand domains but the original code doesn't have any algorythms built into it to rank pages, so a heap of modifications are needed it you want to be able to sift results like other large search engines.
I would imagine that it would be quite fine upto around a hundred domains without to much trouble.
It is inclinde to slwo down searches once around ten thousand pages have been indexed but again a few changes to the way PHPDIG queries the database can speed this up.
In general terms constant cleaning of the index and keyword dictionary will make a huge difference to it overall speed.
A simple clean of the dashes will make it around 10% quicker when it's complete.

My server now runs with eight gig of ram, 3.2gig processors x 2 and an array of fast disk drives enables it to function quite quickly. Early on I put it on two servers because when it was indexing it slowed searches down to a crawl but setting up another server that does the indexing and then passes the data over to the database on the other server speeds things up a heap.
Dave A is offline   Reply With Quote