Spidering not completing

[lentilsoup]

I used PHPDig on my local machine and successfully spidered and searched my website. I then uploaded this code to a webserver and it stopped working properly.

At first it would not find anything. On 'digging' a new site it would say it had completed indexing very quickly and then on return to the admin page it would say 1 host and zero for everything else (keywords etc..) in the statistics. After fiddling around I eventually indexed some pages, but not all. I'm sure I didn't really change anything. It seems a little sporadic.

So the problem is: the spider doesn't seem to be following all links as it has only found around 9 pages whereas the site has around 100 pages. I change nothing since last night and now it won't find any links again.

Here's a checklist of things I have done based on previous posts and/or the documentation:

- 'safe-mode' is off: Check
- 'allow_url_fopen' is on: Check
- Site is linked up entirely by simple <a href> tags: Check
- Tried the DEFINES as follows:
define('ABSOLUTE_SCRIPT_PATH',correct with no trailing slash);
define('LIMIT_TO_DIRECTORY',false);
define('PHPDIG_SESSID_REMOVE',true);
define('PHPDIG_IN_DOMAIN',true);
Check
- The three directories that require it are CHMOD 777: Check
- Tried the $pu['port'] = ''; bugfix - both as '' and '0': Check

Could this be something to do with the commercial server I am running this on? Or is it more likely the code?

[Dave A]

This could be a problem with the way the new sever balances it loading, some odd things have turned up depending on how the laod balancing is set up

[lentilsoup]

Is there any way around this if I have no control over the server myself?