View Single Post
Old 11-12-2005, 11:55 AM   #1
galacticvoyager
Green Mole
 
Join Date: Nov 2005
Posts: 3
robots.txt seems to be ignored :?

While PHPDIG is spidering my site it reports the follow:

48:http://www.galacticvoyager.com/bb/phpBB2/index.php
(time : 00:00:37)
+ + + + + + + + + + + + + + + + + + + + +

I thought this meant it was indexing the given URL however my robots.txt file look like this:

User-agent: *
disallow: /bb/
disallow: /pat/
disallow: /mod/

Shouldn't the url with /bb/ be screen out of the indexing process or am I missing something about the relationship to the robots.txt flle?

Thanks for your help.
galacticvoyager is offline   Reply With Quote