PDA

View Full Version : robots.txt versus robotsxx.txt


Charter
03-11-2004, 06:00 PM
Hi. It's one thing to not parse a robots.txt file correctly, but it's another thing to ask for a nonstandard filename.

218.232.120.81 - - [10/Mar/2004:22:36:20 -0800] "GET /robotsxx.txt HTTP/1.0" 404 0 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"

218.232.120.81 - - [10/Mar/2004:22:36:21 -0800] "GET / HTTP/1.0" 200 37469 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"