PhpDig.net - View Single Post

Charter · 12-28-2003, 02:27 PM

Hi. PhpDig is restrictive when it parses a robots.txt file. Try applying the code in this thread and then set the robots.txt file as so:

Code:

User-agent: PhpDig
Disallow:

User-agent: *
Disallow: /

After a crawl, you can delete/exclude directories from the admin panel. Also, does the hang always happen, and what entries are in the tempspider table?

12-28-2003, 02:27 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. PhpDig is restrictive when it parses a robots.txt file. Try applying the code in this thread and then set the robots.txt file as so: Code: User-agent: PhpDig Disallow: User-agent: * Disallow: / After a crawl, you can delete/exclude directories from the admin panel. Also, does the hang always happen, and what entries are in the tempspider table? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.