Searching *.txt files
 

The site url is 'http://www.rafars.org/'.

The PhpDig search url is 'http://cgi.rafars.org/php/S/search.php'

I have one particular page (www.rafars.org/archives/packet.html) with links to *.txt files.

None of the *.txt files get indexed - any ideas?

Hi. What level did you select when you made the crawl? You might try increasing the level to index these pages.

I used 20.

The answer to the problem is:-

<meta name="ROBOTS" content="index,nofollow"> is wrong?
I set it to <meta name="ROBOTS" content="index,follow"> and now the *.txt files are found.

Why didn't I think of that!


Unfortunately this has shown up another funny!

PhpDig reported that it found 25 *.txt files but I have a total of 67.

Hi. The page http://www.rafars.org/archives/packet.html does not appear to link to all of the text files, as it only links to text files from years 2003 and 2002. If there are no links to the other text files, PhpDig will not find them.

Hi.
There are links to every file.

Whilst looking for the reason why only 2003 and 2002 were indexed I saw that after the last *.txt file in 2002 there was a link to an *.exe file.

Adding the phpdig exclude and include code around each *.exe file enabled PhpDig to find all the *.txt files.

ttfn