spidering page not found

[merrittholmes]

hi,

When I add a URL it tries to go to spider.php but says it can't find it. The file is definately there and I have CHMOD it to 777.

Any ideas?

I have openened it and it is not corrupt also

[vinyl-junkie]

Welcome to the forum, merrittholmes. :D

Several questions come to mind:

What is the URL you are trying to spider?

Exactly how did go about trying to spider it? In other words, was it a single URL you were trying to add to your index, or just something you think should have been indexed?

Do you have any robots exclusion that would prevent it from being indexed?

Do you have phpdig setup so that you can only re-spider every 7 day, for example?

Answers to these questions might clear up the mystery.

[merrittholmes]

Hi,

Thanks for the reply. In answer to your questions.


I have tried a few URLs. One internal (http://www.expw.co.uk), and also http://www.expw.co.uk/newsite/myweb/www/expw-consulting.php?id=1 and also one external (http://www.chaplinspantos.co.uk). Each time I add the URL on the admin page and then choose dig It (I have tried a variety of depth options too) it just comes up with the 'Page cannot be displayed' page when it automatically tries to call spider.php . I haev checked the permissions on the file (777) and also that it is not corrupt (i have opened it in a text editor).

I have also tried calling the spider.php script directly in the URL box.

There is no exclusions in Robots.txt

It seems to be picked up if I copy it to the directory below (but then obviously I get lots of errors). I have checked the permissions of both file and directory.

Any help would be great. I'm happy to send screenshots if needed.

[vinyl-junkie]

I was able to spider a couple of pages from your site, so I think it might be a problem with the settings on your server.

Check your phpinfo page to make sure safe_mode is off and allow_url_fopen is on. To find out how those are set on your server, just run the following script:<?php phpinfo(); ?>
Also, have a look at this thread (http://www.phpdig.net/forum/showthread.php?t=1118&highlight=safe+mode).

Hope this helps. :)

[merrittholmes]

again thanks for the quick reply,

as you can see from http://www.expw.co.uk/phpinfo.php both settings are correct.

Could it be possible that the spider.php script is expecting a parameter that is not getting passed? I'm so out of ideas.

Mike