spidering page not found

merrittholmes · 09-10-2004, 08:20 AM

hi,

When I add a URL it tries to go to spider.php but says it can't find it. The file is definately there and I have CHMOD it to 777.

Any ideas?

I have openened it and it is not corrupt also

vinyl-junkie · 09-10-2004, 09:56 AM

Welcome to the forum, merrittholmes.

Several questions come to mind:

What is the URL you are trying to spider?

Exactly how did go about trying to spider it? In other words, was it a single URL you were trying to add to your index, or just something you think should have been indexed?

Do you have any robots exclusion that would prevent it from being indexed?

Do you have phpdig setup so that you can only re-spider every 7 day, for example?

Answers to these questions might clear up the mystery.

merrittholmes · 09-13-2004, 02:48 AM

Hi,

Thanks for the reply. In answer to your questions.

I have tried a few URLs. One internal (http://www.expw.co.uk), and also http://www.expw.co.uk/newsite/myweb/...lting.php?id=1 and also one external (http://www.chaplinspantos.co.uk). Each time I add the URL on the admin page and then choose dig It (I have tried a variety of depth options too) it just comes up with the 'Page cannot be displayed' page when it automatically tries to call spider.php . I haev checked the permissions on the file (777) and also that it is not corrupt (i have opened it in a text editor).

I have also tried calling the spider.php script directly in the URL box.

There is no exclusions in Robots.txt

It seems to be picked up if I copy it to the directory below (but then obviously I get lots of errors). I have checked the permissions of both file and directory.

Any help would be great. I'm happy to send screenshots if needed.

vinyl-junkie · 09-13-2004, 04:47 AM

I was able to spider a couple of pages from your site, so I think it might be a problem with the settings on your server.

Check your phpinfo page to make sure safe_mode is off and allow_url_fopen is on. To find out how those are set on your server, just run the following script:

Code:

<?php phpinfo(); ?>

Also, have a look at this thread.

Hope this helps.

merrittholmes · 09-13-2004, 07:26 AM

again thanks for the quick reply,

as you can see from http://www.expw.co.uk/phpinfo.php both settings are correct.

Could it be possible that the spider.php script is expecting a parameter that is not getting passed? I'm so out of ideas.

Mike

09-10-2004, 08:20 AM	#1
merrittholmes Green Mole Join Date: Sep 2004 Posts: 3	spidering page not found hi, When I add a URL it tries to go to spider.php but says it can't find it. The file is definately there and I have CHMOD it to 777. Any ideas? I have openened it and it is not corrupt also

09-13-2004, 02:48 AM	#3
merrittholmes Green Mole Join Date: Sep 2004 Posts: 3	spidering page not found Hi, Thanks for the reply. In answer to your questions. I have tried a few URLs. One internal (http://www.expw.co.uk), and also http://www.expw.co.uk/newsite/myweb/...lting.php?id=1 and also one external (http://www.chaplinspantos.co.uk). Each time I add the URL on the admin page and then choose dig It (I have tried a variety of depth options too) it just comes up with the 'Page cannot be displayed' page when it automatically tries to call spider.php . I haev checked the permissions on the file (777) and also that it is not corrupt (i have opened it in a text editor). I have also tried calling the spider.php script directly in the URL box. There is no exclusions in Robots.txt It seems to be picked up if I copy it to the directory below (but then obviously I get lots of errors). I have checked the permissions of both file and directory. Any help would be great. I'm happy to send screenshots if needed.

09-13-2004, 04:47 AM	#4
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	I was able to spider a couple of pages from your site, so I think it might be a problem with the settings on your server. Check your phpinfo page to make sure safe_mode is off and allow_url_fopen is on. To find out how those are set on your server, just run the following script: Code: <?php phpinfo(); ?> Also, have a look at this thread. Hope this helps.

09-13-2004, 07:26 AM	#5
merrittholmes Green Mole Join Date: Sep 2004 Posts: 3	spidering page not found again thanks for the quick reply, as you can see from http://www.expw.co.uk/phpinfo.php both settings are correct. Could it be possible that the spider.php script is expecting a parameter that is not getting passed? I'm so out of ideas. Mike

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
not spidering all pages (too many links on page?)	mirdin	Troubleshooting	2	09-01-2004 06:08 AM
Spidering....links found : 0	-IAN-	Troubleshooting	23	08-25-2004 04:53 PM
Speed up spidering by skipping internal page links	peterpeter	Mod Requests	6	04-18-2004 07:17 AM
Spidering with "links found : 0"	fransdars	Troubleshooting	4	02-02-2004 12:03 AM
no link found... again	moutonrebelle	Troubleshooting	3	12-30-2003 03:12 AM

09-10-2004, 09:56 AM	#2
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Welcome to the forum, merrittholmes. Several questions come to mind: What is the URL you are trying to spider? Exactly how did go about trying to spider it? In other words, was it a single URL you were trying to add to your index, or just something you think should have been indexed? Do you have any robots exclusion that would prevent it from being indexed? Do you have phpdig setup so that you can only re-spider every 7 day, for example? Answers to these questions might clear up the mystery.