PhpDig.net - View Single Post - 1.8.4 spider doesn't handle some files :(

Juri Savin · 12-15-2004, 07:14 AM

So, first a upgraded to 1.8.5, got improved security and the same problem. Then I read the post you adviced me (thank you). But as a PHP-programmer I don't quite understand, what did the author of that post meaned.

PhpDig works via http requests like a usual web browser. It doesn't read files directly via ftp. So, there is no difference to PhpDig how the link was generated. Because PhpDig can see just the output of web server, not source code.

Today I used my rest chance - I converted all trouble links into virtual files (mod_rewrite). Now the links looks like <a href="/catalogue/import/111.html"></a>. And... It does't work again!

Over and over again I get this:

<citation>
SITE : http://www.loip.ru/
Exclude paths :
- @NONE@
1:http://www.loip.ru/catalogue/import/
(time : 00:00:06)
No link in temporary table
</citation>

The page I tried to index is here: http://www.loip.ru/catalogue/import/

Maybe a bug in the html source? Or in PhpDig engine?

12-15-2004, 07:14 AM	#3
Juri Savin Green Mole Join Date: Dec 2004 Posts: 4	So, first a upgraded to 1.8.5, got improved security and the same problem. Then I read the post you adviced me (thank you). But as a PHP-programmer I don't quite understand, what did the author of that post meaned. PhpDig works via http requests like a usual web browser. It doesn't read files directly via ftp. So, there is no difference to PhpDig how the link was generated. Because PhpDig can see just the output of web server, not source code. Today I used my rest chance - I converted all trouble links into virtual files (mod_rewrite). Now the links looks like <a href="/catalogue/import/111.html"></a>. And... It does't work again! Over and over again I get this: <citation> SITE : http://www.loip.ru/ Exclude paths : - @NONE@ 1:http://www.loip.ru/catalogue/import/ (time : 00:00:06) No link in temporary table </citation> The page I tried to index is here: http://www.loip.ru/catalogue/import/ Maybe a bug in the html source? Or in PhpDig engine?