PhpDig.net - View Single Post

tryangle · 05-02-2004, 09:22 AM

Hi vinyl-junkie,

Thanks for your reply.

I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source:

<meta name="robots" content="index,follow">

Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.

05-02-2004, 09:22 AM	#3
tryangle Green Mole Join Date: Jan 2004 Location: Penns Woods Posts: 17	Search Depth Hi vinyl-junkie, Thanks for your reply. I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source: <meta name="robots" content="index,follow"> Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.