|
Three things I can think of...
1) The links may not match the regex for links. Search for ([a-z]{3,5}://) in the robot_functions.php file to find two regex for links.
2) Some of the pages you are trying to crawl are encoded windows-1251 but the search results look to be using iso-8859-1 instead.
3) Some of the pages are using a whole lot of HTML entities instead of an encoding. PhpDig currently support windows-1251 for Cyrillic.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
|