PhpDig.net - View Single Post

alivin70 · 12-18-2003, 08:49 AM

I tried to index a subdirectory of a website, for example

ww.mysite.com/mydir/

but the spider goes to other dirs if it find the link. For example it can find the link to

ww.mysite.com

and starts to index the whole site.

Do you think it's possible to avoid that?

12-18-2003, 08:49 AM	#1
alivin70 Orange Mole Join Date: Sep 2003 Posts: 40	Indexing subdirectory I tried to index a subdirectory of a website, for example ww.mysite.com/mydir/ but the spider goes to other dirs if it find the link. For example it can find the link to ww.mysite.com and starts to index the whole site. Do you think it's possible to avoid that?