PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 12-02-2003, 07:09 AM   #1
rafarspd
Orange Mole
 
rafarspd's Avatar
 
Join Date: Nov 2003
Location: High Wycombe, Bucks. UK
Posts: 35
Question Searching *.txt files

The site url is 'http://www.rafars.org/'.

The PhpDig search url is 'http://cgi.rafars.org/php/S/search.php'

I have one particular page (www.rafars.org/archives/packet.html) with links to *.txt files.

None of the *.txt files get indexed - any ideas?
__________________
Regards
Bob Finch
rafarspd is offline   Reply With Quote
Old 12-02-2003, 07:27 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. What level did you select when you made the crawl? You might try increasing the level to index these pages.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 12-04-2003, 03:20 AM   #3
rafarspd
Orange Mole
 
rafarspd's Avatar
 
Join Date: Nov 2003
Location: High Wycombe, Bucks. UK
Posts: 35
I used 20.

The answer to the problem is:-

<meta name="ROBOTS" content="index,nofollow"> is wrong?
I set it to <meta name="ROBOTS" content="index,follow"> and now the *.txt files are found.

Why didn't I think of that!


Unfortunately this has shown up another funny!

PhpDig reported that it found 25 *.txt files but I have a total of 67.
__________________
Regards
Bob Finch
rafarspd is offline   Reply With Quote
Old 12-04-2003, 08:25 AM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. The page http://www.rafars.org/archives/packet.html does not appear to link to all of the text files, as it only links to text files from years 2003 and 2002. If there are no links to the other text files, PhpDig will not find them.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 12-05-2003, 09:01 AM   #5
rafarspd
Orange Mole
 
rafarspd's Avatar
 
Join Date: Nov 2003
Location: High Wycombe, Bucks. UK
Posts: 35
Hi.
There are links to every file.

Whilst looking for the reason why only 2003 and 2002 were indexed I saw that after the last *.txt file in 2002 there was a link to an *.exe file.

Adding the phpdig exclude and include code around each *.exe file enabled PhpDig to find all the *.txt files.

ttfn
__________________
Regards
Bob Finch
rafarspd is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
searching PDF files bcunico External Binaries 3 02-24-2006 01:40 AM
indexing *.txt files skunkone Troubleshooting 2 11-08-2004 07:03 AM
Why does phpDig rely on creating txt files for descriptions? misterbearcom How-to Forum 1 06-13-2004 11:09 PM
robots.txt versus robotsxx.txt Charter IPs, SEs, & UAs 0 03-11-2004 06:00 PM


All times are GMT -8. The time now is 02:45 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.