PhpDig.net - View Single Post

Charter · 11-06-2005, 09:44 PM

So far I've found that one of the better ways to tell if someone is trying to index his/her site, where both the site and PhpDig are installed on a load balanced server, is to look at the stats on the PhpDig main admin page after an attempted index:

Code:

DataBase status
Hosts :            1 Entries
Pages :            0 Entries
Index :            0 Entries
Keywords :         0 Entries
Temporary table :  0 Entries

Sometimes people on load balanced servers report that they are able to index other sites, but not their own, so the zeros above are replaced with greater than zero numbers. Basically PhpDig and some other such spider search combos bite, in their current form, when it comes to load balancing.

Here's a really basic history: The way it worked, way back when, is that each site had its own number, but with web growth, there needed to be a way to deal with all the webpage requests. Fast forward, and now there are clusters of web servers that have their own number, and some software exists to process the number of requests.

In short, that means, while you can call http://www.yoursite.com from a browser, there's some, erm, mystery that goes on for that to happen, and it no longer depends on a unique number for each site. Rather, it's a unique number for each cluster of servers, and that, uh, mystery directs browser requests where they are to go.

Load balancing takes webpage requests, using something in the realm of mystery, and directs requests to the best, i.e., less busy server, and feeds the webpage you see onscreen. This process makes it impossible for PhpDig and some other such spider search combos, in their current form, to find where they are supposed to go to do their stuff.

11-06-2005, 09:44 PM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	So far I've found that one of the better ways to tell if someone is trying to index his/her site, where both the site and PhpDig are installed on a load balanced server, is to look at the stats on the PhpDig main admin page after an attempted index: Code: DataBase status Hosts : 1 Entries Pages : 0 Entries Index : 0 Entries Keywords : 0 Entries Temporary table : 0 Entries Sometimes people on load balanced servers report that they are able to index other sites, but not their own, so the zeros above are replaced with greater than zero numbers. Basically PhpDig and some other such spider search combos bite, in their current form, when it comes to load balancing. Here's a really basic history: The way it worked, way back when, is that each site had its own number, but with web growth, there needed to be a way to deal with all the webpage requests. Fast forward, and now there are clusters of web servers that have their own number, and some software exists to process the number of requests. In short, that means, while you can call http://www.yoursite.com from a browser, there's some, erm, mystery that goes on for that to happen, and it no longer depends on a unique number for each site. Rather, it's a unique number for each cluster of servers, and that, uh, mystery directs browser requests where they are to go. Load balancing takes webpage requests, using something in the realm of mystery, and directs requests to the best, i.e., less busy server, and feeds the webpage you see onscreen. This process makes it impossible for PhpDig and some other such spider search combos, in their current form, to find where they are supposed to go to do their stuff. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.