|
01-31-2005, 08:10 AM | #1 |
Orange Mole
Join Date: Dec 2004
Location: Tennessee
Posts: 60
|
pages indexed
So, how many pages have people indexed? NewTangent and myself have done about 30,000 pages with our installation of PHPDig, and we are wondering what other people have indexed (anyone over a million??)
JMitchell
__________________
60,000 pages indexed!!!!! http://www.sharemylink.com |
01-31-2005, 11:07 AM | #2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
I index just enough pages to test PhpDig or run the online demo, nowhere near 30,000 pages.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
02-01-2005, 11:55 AM | #3 |
Orange Mole
Join Date: Dec 2004
Location: Tennessee
Posts: 60
|
ok, I was wondering about phpdig speeds when you have a huge database, and if there was a way to have several databases, and phpdig searches through them, that way I could have multiple databases on multiple servers, thereby maximizing my speed - is this correct thinking, and can it be done?
__________________
60,000 pages indexed!!!!! http://www.sharemylink.com |
02-02-2005, 11:13 AM | #4 |
Orange Mole
Join Date: Dec 2004
Location: Tennessee
Posts: 60
|
I also remember that others were talking about huge databases a few weeks/months ago, and was wondering if they had any results.
jmitchell
__________________
60,000 pages indexed!!!!! http://www.sharemylink.com |
02-03-2005, 02:37 AM | #5 |
Purple Mole
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
|
Size of database
My database stands at
Hosts : 105050 Entries Pages : 3430752 Entries Index : 14938699 Entries Keywords : 392470 Entries Temporary table : 0 Entries Which is around 280mb of web space used but I am aware that some form of clipping may occur when the host free space starts to get small, so I increase it by another fifty mb. Search times have shortened when I have recently gone over to braodband. I have found that files left in the temp spider clog up the stsem and slow searches down a heap. |
02-03-2005, 11:05 AM | #6 |
Orange Mole
Join Date: Dec 2004
Location: Tennessee
Posts: 60
|
whats your search site Dave?
I'd like to see how you are using it. JMitchell
__________________
60,000 pages indexed!!!!! http://www.sharemylink.com |
02-12-2005, 11:49 AM | #7 |
Purple Mole
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
|
The problem starts when the search times start to be extended, it would appear that Google got around the problem by using an cluster of computers each containing some of the database, So I would imagine they have one front end machine that could lead the data on to machines that have smaller databases, maybe one database that covers A-b entries then another for C-D entries so each one only carries some of the load.
Over four billion Web pages, each an average of 10KB, all fully indexed. Up to 2,000 PCs in a cluster. Over 30 clusters. 104 interface languages including Klingon and Tagalog. One petabyte of data in a cluster -- so much that hard disk error rates of 10-15 begin to be a real issue. Sustained transfer rates of 2Gbps in a cluster. An expectation that two machines will fail every day in each of the larger clusters. No complete system failure since February 2000. More information can be found at: http://www.zdnet.com.au/insight/soft...9168647,00.htm It is a good read to see how they have managed to get around the same problems we are facing. On a passing note it is cloudy this morning in New Zealand and if the sunburn times are less than twenty minutes I may have to increase my search capacity and see if I can catch some fish. Never forget that running a search engine needs time spent searching out ever greater places where someone can extend the fishing rod, back of the boat, a glass of wine, indexing the ocean is a really important thing... It helps you find common answers to heaps of searches... Grin.. |
02-13-2005, 03:03 AM | #8 |
Purple Mole
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
|
Hi please check out my user profile for my search site and contact details.
This forum isn't the correct place to post things that can be indexed by major search engines and may appear to be adverts. Grin...! So please view my user profile, my contact details are available. I am more than glad to assist anyone using this software to the best of my ability. Many regards Dave A |
02-15-2005, 01:23 PM | #9 | |
Purple Mole
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
|
Quote:
|
|
Thread Tools | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Spider stops before all pages are indexed | halide | Troubleshooting | 3 | 07-19-2005 01:26 AM |
Too few pages indexed, Umlaut problem | salzbermat | Troubleshooting | 4 | 12-16-2004 11:00 AM |
Pages not re-indexed | wx3 | Troubleshooting | 0 | 09-16-2004 06:53 PM |
Number of pages indexed | claudiomet | How-to Forum | 0 | 08-30-2004 03:26 PM |
how to index only not indexed pages? | zaartix | How-to Forum | 2 | 07-14-2004 05:23 AM |