phpdig seems to guess some urls and spider it
hi!
my urls look like this: domain.com/dir1/dir2/something
now phpdig spiders them fine, all right. but it also seems to "guess" new urls. i saw it spidering domain.com/dir1/dir2/ although that isn't linked anywhere.
why is that and how can i stop this?
|