Spider Crawl stopping too early
 

I've got a fresh install of phpdig on my server, the site's links are all properly linked and should be totally accessible, and I've tried numerous combinations on the search depth and links per, even as far as modifying the config to allow me a search and links setting of 100 each. But so far the absolute best I have been able to manage to index is less than 40 pages, on a site that should be returning several hundred.

What's causing it to just stop, and is there any way I can fix this or get around it and have it index the site properly?

Set LIMIT_TO_DIRECTORY to false, set PHPDIG_IN_DOMAIN to true, set "search depth" to a large number, set "links per" to zero, choose the "no" option.

Done all of that, I've even got search depth set to 1000. The thing stops at 51 each and every time and there are definately pages not being indexed. I've tried god knows how many variations here and still I cannot get past 51.

When it stops, do you see an "indexing complete" message onscreen?

Nope, I do not... and I have tried running this in both IE, and my default browser Firefox.

Firefox and IE should both be able to handle more than 51 links shown onscreen. Is your PHP set with safe mode on or off? If safe mode is off, does your host have a script set to kill certain processes?