HTTP/1.1 404 Not Found
 

Hi,

Spidering stays in progress. Website stays locked. No indexing occurs. Same result for main site and virtual hosts on localhost. IPs and domain names set in HOSTS (Windows XP). What could be wrong?

Thxs,

Schop

Spidering in progress...

HTTP/1.1 404 Not Found
HTTP/1.1 200 OK
Date: Sat, 10 Apr 2004 09:55:34 GMT
Server: Apache/2.0.49 (Win32) PHP/4.3.4
Last-Modified: Wed, 31 Mar 2004 10:31:12 GMT
ETag: "2819-a31f-d86f5c00"
Accept-Ranges: bytes
Content-Length: 41759
Content-Type: text/html; charset=ISO-8859-1

HTTP/1.1 404 Not Found

SITE : http://www.gidge1.info/

Exclude paths :
- @NONE@
HTTP/1.1 200 OK
Date: Sat, 10 Apr 2004 09:55:39 GMT
Server: Apache/2.0.49 (Win32) PHP/4.3.4
Last-Modified: Wed, 31 Mar 2004 10:31:12 GMT
ETag: "2819-a31f-d86f5c00"
Accept-Ranges: bytes
Content-Length: 41759
Content-Type: text/html; charset=ISO-8859-1

Welcome to the forum, schop. We're glad you could join us!

Click on the domain name that you posted, and you'll get a "page cannot be displayed" error. Most likely, that means you typed the URL incorrectly or that the site is down. Looks like an invalid URL to me.

Try clearing out your phpDig tables and spidering again. Before you do that though, make sure you have the value of LIMIT_DAYS in config.php set to a value where you can re-spider the site now.

Hope this helps. :)

404 Not Found persists
 

Hi Pat,

Thanks for your reply. The site www.gidge1.info is a virtual host on my local Apache server.

I have followed your suggestions but the problem persists, even when I try to index the local primary host or one of my published websites.

phpDig seems to find the first htm-file to be indexed as it returns the correct number of bytes. So now I wonder what exactly triggers the 404 error message.

All suggestions for resolving this problem welcome!

Remy

Have you tried indexing from shell? This thread might be of some assistance with that.

Hi. The 404 might be because there is no robots.txt file - not needed to run PhpDig - it just returns a 404 if not found. Are you using base href tags in your pages? If so, check this thread.

404 Not Found
 

Hi, Have implemented the suggestions in both threads but the result stays the same. Even tested a very basic html file with only a title in the head and a short text in the body. Same result.

If the 404 error would be caused by the missing robots.txt file, where should it be placed in the tree and what content should it have if any?

Also had a closer look at connect.php and config.php. But there are no problems with the database connection and I couldn't see anything wrong with config.php. (BTW, where is $relative_script_path initially set?) Also, php safe_mode is Off.

Which settings are the most likely candidate for the indexing and spidering process to be aborted: Apache, php or phpDig?

Remy

Hi. The robots.txt file is not needed for PhpDig to run, but if you want such a file just stick it in web root. The $relative_script_path variable is defined is several locations, just do a file search. BTW, how many links off the main page are present?

Hi, Placing robots.txt in the web root got rid of the 404 error. Thanks!

Now it gets to: Fatal error: Call to undefined function: is_executable() in C:\Program Files\Apache Group\Apache2\htdocs\phpDig\admin\robot_functions.php on line 685

Any clues?

Remy

This thread explains just what you need to do about that. Hope it helps. :)

Hi,

Actually found the solution myself: Set USE_IS_EXECUTABLE_COMMAND to 0 in config.php.

Will later test with setting it to 1 and defining the external binaries.

Thanks for all the help. Much appreciated!

Remy