View Single Post
Old 10-13-2006, 10:21 PM   #5
Dave A
Purple Mole
 
Dave A's Avatar
 
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
You should find that it will stop the spider from finding links outside the domain and it also stps the spider in most cases find sub domains as well.
I often index geaneology based websites and they have heaps of free pages set as sub domains before I tried those setting it would alays try and index the whole thing.

So web developers do add a few html comments inside a domain which can trapa spider into indexing onwards but in the end I wrote a small patch that stops this.
Not may web developers have the skilss needed to try trick spiders Google may not revisit to often if it is found.

Keaap up the good work!

My version which has been heaviliy modified has now indexed over one hundred and ten million webpages.
So the scope to to enhance PHPDIG is very good to say the least.
Dave A is offline   Reply With Quote