PDA

View Full Version : Nothing indexed on some sites


tryangle
04-30-2004, 06:01 PM
Hi,

Sometimes I get output like this after I try to index a site:

SITE : http://www.domain.tld/
Exclude paths :
- @NONE@
1:http://www.domain.tld/
(time : 00:00:47)
No link in temporary table

I checked the source of the site's index page and found this:

<meta name="robots" content="index,follow">

How come I don't get anything though?

TIA for any info on this...

vinyl-junkie
04-30-2004, 07:27 PM
What search depth did you choose? If you left the default at zero, phpDig will only spider the root.

tryangle
05-02-2004, 10:22 AM
Hi vinyl-junkie,

Thanks for your reply.

I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source:

<meta name="robots" content="index,follow">

Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.

vinyl-junkie
05-02-2004, 01:31 PM
Perhaps this thread (http://www.phpdig.net/showthread.php?s=&threadid=127&highlight=index) might be of some help.