Nothing indexed on some sites
 

Hi,

Sometimes I get output like this after I try to index a site:

SITE : http://www.domain.tld/
Exclude paths :
- @NONE@
1:http://www.domain.tld/
(time : 00:00:47)
No link in temporary table

I checked the source of the site's index page and found this:

<meta name="robots" content="index,follow">

How come I don't get anything though?

TIA for any info on this...

What search depth did you choose? If you left the default at zero, phpDig will only spider the root.

Search Depth
 

Hi vinyl-junkie,

Thanks for your reply.

I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source:

<meta name="robots" content="index,follow">

Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.

Perhaps this thread might be of some help.