Quote:
Originally posted by Charter
Hi. PhpDig is restrictive when it parses a robots.txt file. Try applying the code in this thread and then set the robots.txt file as so:
Code:
User-agent: PhpDig
Disallow:
User-agent: *
Disallow: /
After a crawl, you can delete/exclude directories from the admin panel. Also, does the hang always happen, and what entries are in the tempspider table?
|
Hi, firstly thanks for the speedy reply!
I've changed the code as suggested in the thread you linked to and modified the robots.txt file as you said, and am getting the same problem each time .. namely that spider.php freezes during the indexing process and locks the site while not indexing any further. I should also mention I have tried completely removing the robots.txt file with no success.
As for the tempspider table, here is the phpMyAdmin dumps in
csv and
xml