PhpDig.net - View Single Post - Can't get PHPDig to index an htaccess protected site

mlerch@mac.com · 02-23-2004, 02:29 PM

Ok... I took a look at the logs. The error log that is. Here is what I get when I try to access that URL directly:

[Mon Feb 23 15:25:53 2004] [error] [client xxx.xxx.xxx.xxx] File does not exist: /Path/to/Web/Server/dir/html/robots.txt

Nothing in the access logs.

Why is it looking for robots.txt?

The access log gives me:

xxx.xxx.xxx.xxx - - [23/Feb/2004:15:32:10 -0800] "HEAD /index.php HTTP/1.1" 302 - "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)"

02-23-2004, 02:29 PM	#23
mlerch@mac.com Green Mole Join Date: Feb 2004 Location: North Las Vegas, Nevada Posts: 18	Ok... I took a look at the logs. The error log that is. Here is what I get when I try to access that URL directly: [Mon Feb 23 15:25:53 2004] [error] [client xxx.xxx.xxx.xxx] File does not exist: /Path/to/Web/Server/dir/html/robots.txt Nothing in the access logs. Why is it looking for robots.txt? The access log gives me: xxx.xxx.xxx.xxx - - [23/Feb/2004:15:32:10 -0800] "HEAD /index.php HTTP/1.1" 302 - "-" "PhpDig/1.8.0 (+http://www.phpdig.net/robot.php)" Last edited by mlerch@mac.com; 02-23-2004 at 02:37 PM.