PhpDig.net - View Single Post

i_am_cam · 12-28-2003, 12:50 PM

Hey, i'm having a problem trying to spider my site. I've read through the other forum topics that match my symptoms but while the blame mostly is aimed at safe_mode being on, my host has it off.

Basically when i'm running spider.php it starts indexing as i'd expect but then hangs after about 10seconds with IE displaying the 'done' alert in the status bar. On the admin index page it says the page is locked as the spider is still running, but nothing else is added.

I downloaded the latest version of phpdig (1.6.5 i believe) and am on PHP Version 4.2.3 (to see my PHP settings if it helps, look here).

The work the spider manages before the hanging bug is what i'd expect .. i can search the pages it's indexed and am pleased with the results, just I need the whole site done!

The search results page can be found here.

Thanks for any help

--Edit--

I've just added this screenshot of what happens while running spider.php, in case this is of any help:

Spider.php Screenshot

--2nd Edit--

And thought i'd add my robots.txt file too

User-agent: PhpDig
Disallow: /forum
Disallow: /phpMyAdmin
Disallow: /sql
Disallow: /templates
Disallow: /templates_c
Allow: /forum/index.php

User-agent: *
Disallow: /

12-28-2003, 12:50 PM	#1
i_am_cam Green Mole Join Date: Dec 2003 Posts: 6	Spider Problem Hey, i'm having a problem trying to spider my site. I've read through the other forum topics that match my symptoms but while the blame mostly is aimed at safe_mode being on, my host has it off. Basically when i'm running spider.php it starts indexing as i'd expect but then hangs after about 10seconds with IE displaying the 'done' alert in the status bar. On the admin index page it says the page is locked as the spider is still running, but nothing else is added. I downloaded the latest version of phpdig (1.6.5 i believe) and am on PHP Version 4.2.3 (to see my PHP settings if it helps, look here). The work the spider manages before the hanging bug is what i'd expect .. i can search the pages it's indexed and am pleased with the results, just I need the whole site done! The search results page can be found here. Thanks for any help --Edit-- I've just added this screenshot of what happens while running spider.php, in case this is of any help: Spider.php Screenshot --2nd Edit-- And thought i'd add my robots.txt file too User-agent: PhpDig Disallow: /forum Disallow: /phpMyAdmin Disallow: /sql Disallow: /templates Disallow: /templates_c Allow: /forum/index.php User-agent: * Disallow: / Last edited by i_am_cam; 12-28-2003 at 01:07 PM.