While PHPDIG is spidering my site it reports the follow:
48:
http://www.galacticvoyager.com/bb/phpBB2/index.php
(time : 00:00:37)
+ + + + + + + + + + + + + + + + + + + + +
I thought this meant it was indexing the given URL however my robots.txt file look like this:
User-agent: *
disallow: /bb/
disallow: /pat/
disallow: /mod/
Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle?
Thanks for your help.