View Single Post
Old 03-11-2004, 07:00 PM   #1
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
robots.txt versus robotsxx.txt

Hi. It's one thing to not parse a robots.txt file correctly, but it's another thing to ask for a nonstandard filename.

218.232.120.81 - - [10/Mar/2004:22:36:20 -0800] "GET /robotsxx.txt HTTP/1.0" 404 0 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"

218.232.120.81 - - [10/Mar/2004:22:36:21 -0800] "GET / HTTP/1.0" 200 37469 "-" "PlantyNet_WebRobot_V1.9 dhkang@plantynet.com"
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote