Thanks, it help a bit but I now have to confess to the main reason for the problem.
I forgot a basic principle ? 
If you want PHPDIG to find a file then it must have a URL somewhere pointing to it.
I had removed some references to older text files from a page.
Problem is now cured by adding the URL's (with no text) at the bottom of the html just above </body></html>.
They are not seen when viewing but PHPDIG finds and indexes them.