Try setting PHPDIG_IN_DOMAIN to true, LIMIT_TO_DIRECTORY to false, both in the config file, and then from the admin panel, use a large search depth, set links per to zero, and choose the no option. You can increase search depth beyond twenty by editing SPIDER_MAX_LIMIT in the config file. Once indexing completes, you should see an 'indexing complete' message onscreen. If, when indexing PDFs, the process seems to die in the middle, it might be a memory issue like in
this thread.