Hi,
on some sites i get this error:
HTTP/1.1 404 Not Found See
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
HTTP/1.1 404 Not Found See
http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
between the "Spidering in progress..." and the spidering itself
(i tried it with many domains - on about the half sites such errors apper)
as example 1 domain:
http://www.otaku-forum.net/