PhpDig.net - robots.txt versus robotsxx.txt

PhpDig.net (http://www.phpdig.net/forum/index.php)

- IPs, SEs, & UAs (http://www.phpdig.net/forum/forumdisplay.php?f=35)

- - robots.txt versus robotsxx.txt (http://www.phpdig.net/forum/showthread.php?t=648)

Charter

03-11-2004 06:00 PM

robots.txt versus robotsxx.txt

Hi. It's one thing to not parse a robots.txt file correctly, but it's another thing to ask for a nonstandard filename.

218.232.120.81 - - [10/Mar/2004:22:36:20 -0800] "GET /robotsxx.txt HTTP/1.0" 404 0 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"

218.232.120.81 - - [10/Mar/2004:22:36:21 -0800] "GET / HTTP/1.0" 200 37469 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"

All times are GMT -8. The time now is 03:12 AM.