Nothing indexed on some sites

tryangle · 04-30-2004, 05:01 PM

Hi,

Sometimes I get output like this after I try to index a site:

SITE : http://www.domain.tld/
Exclude paths :
- @NONE@
1:http://www.domain.tld/
(time : 00:00:47)
No link in temporary table

I checked the source of the site's index page and found this:

<meta name="robots" content="index,follow">

How come I don't get anything though?

TIA for any info on this...

vinyl-junkie · 04-30-2004, 06:27 PM

What search depth did you choose? If you left the default at zero, phpDig will only spider the root.

tryangle · 05-02-2004, 09:22 AM

Hi vinyl-junkie,

Thanks for your reply.

I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source:

<meta name="robots" content="index,follow">

Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.

vinyl-junkie · 05-02-2004, 12:31 PM

Perhaps this thread might be of some help.

04-30-2004, 05:01 PM	#1
tryangle Green Mole Join Date: Jan 2004 Location: Penns Woods Posts: 17	Nothing indexed on some sites Hi, Sometimes I get output like this after I try to index a site: SITE : http://www.domain.tld/ Exclude paths : - @NONE@ 1:http://www.domain.tld/ (time : 00:00:47) No link in temporary table I checked the source of the site's index page and found this: <meta name="robots" content="index,follow"> How come I don't get anything though? TIA for any info on this... Last edited by tryangle; 04-30-2004 at 05:06 PM.

05-02-2004, 09:22 AM	#3
tryangle Green Mole Join Date: Jan 2004 Location: Penns Woods Posts: 17	Search Depth Hi vinyl-junkie, Thanks for your reply. I've tried a search depth of 1 or 2. But it didn't matter, I got the same results... even though they are not one page sites... and they've got lots of internal and external links. What baffles me is that the information on the home page is not even getting indexed, even though there is this line in their source: <meta name="robots" content="index,follow"> Fortunately I've only gotten this on just a few sites, but I'm wondering what would prevent a site from getting spidered... when it appears to be inviting it.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
List how many Sites have been indexed?	Dan	Mod Requests	1	11-17-2006 06:00 AM
Certain sites, pages and pdfs are not indexed 1.8.9 RC1 [Workaround included]	obottek	Bug Tracker	0	08-24-2006 05:30 AM
show sites indexed	richwilson	How-to Forum	2	04-06-2006 04:31 PM
Meta Robots = NoIndex, or already indexed : No content indexed	jerrywin5	How-to Forum	2	04-06-2005 02:50 PM
Many sites indexed - how to restrict search results to just one or two ?	ciaran@clissman	How-to Forum	2	05-27-2004 01:11 AM

04-30-2004, 06:27 PM	#2
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	What search depth did you choose? If you left the default at zero, phpDig will only spider the root.

05-02-2004, 12:31 PM	#4
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Perhaps this thread might be of some help.