Spidering....links found : 0 - Page 2

Dave A · 08-24-2004, 11:30 AM

links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

When I get this I usually delete the site from the admin page and then go back and respider it using different settings. Most times it works a treat.

But I am a new boy ay using this software...

vinyl-junkie · 08-24-2004, 01:31 PM

Quote:

Originally Posted by rispbiz

There only a few sites that I seem to have this problem with. Strangly enough one of the others off hand is http://www.hotmail.com

I'd be willing to bet that hotmail.com won't allow you to index them. There's a thread here in the forum about something similar, but I'm unable to find it at the moment.

rispbiz · 08-24-2004, 03:08 PM

That could be with hotmail.com but the result is the same with www.hotdial.net which does allow indexing and it is like the spider doesnt even try indexing the url.

Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://www.hotdial.net/
Exclude paths :
-
- @NONE@
No link in temporary table

--------------------------------------------------------------------------------

links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

If possiable could someone try indexing this site with there phpdig and let me know if they have a problem indexing it. Then I would know that if another phpdig site cant index it then it would more than likley be a problem with the url rather than an issue with my spider.

If other phpdig se can index it then I would have to figure out where the problem is in my engine.

Thanks for any help.
2-surf.net

vinyl-junkie · 08-24-2004, 03:53 PM

Quote:

Originally Posted by rispbiz

That could be with hotmail.com but the result is the same with www.hotdial.net which does allow indexing and it is like the spider doesnt even try indexing the url.

I don't know about that. I got the same results as you. I've been helping in the forum for quite a while, and usually if a site can be indexed, I won't have a problem doing so.

rispbiz · 08-24-2004, 04:06 PM

I will work with the webmaster and see if I can get him to change the index page, and then try to reindex. Maybe there is something on the page that php dig doesn't like.

Thank you for trying the url for me and quick responses.

With both of us not being able to index the site leaves a lot of questions. Is it the site or something to do with phpdig. HUH???

Thank You
2-surf.net

vinyl-junkie · 08-24-2004, 05:15 PM

Quote:

Originally Posted by rispbiz

With both of us not being able to index the site leaves a lot of questions. Is it the site or something to do with phpdig. HUH???

I'm betting it's a problem with the site.

Good luck on getting it solved!

rispbiz · 08-25-2004, 09:45 AM

Here is what the problem was, Which makes no sense.

The website had a robot.txt file in the that only had this line.

Disallow: 4.15.191.215

I had webmaster remove the robots.txt and it indexed fine.

How come the Disallow: is causing a problem with the sipder?

Thank You,
2-Surf.net

Charter · 08-25-2004, 10:44 AM

Quote:

Originally Posted by rispbiz

Here is what the problem was, Which makes no sense.

The website had a robot.txt file in the that only had this line.

Disallow: 4.15.191.215

I had webmaster remove the robots.txt and it indexed fine.

How come the Disallow: is causing a problem with the sipder?

Thank You,
2-Surf.net

That is not standard robots.txt format.

vinyl-junkie · 08-25-2004, 04:53 PM

I think they might have been trying to ban someone from visiting their site. Trouble is, you don't do that via robots.txt. You do it via .htaccess.

08-24-2004, 11:30 AM	#16
Dave A Purple Mole Join Date: Aug 2004 Location: North Island New Zealand Posts: 170	O files found. links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! When I get this I usually delete the site from the admin page and then go back and respider it using different settings. Most times it works a treat. But I am a new boy ay using this software...

08-24-2004, 04:06 PM	#20
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	Very Strange! I will work with the webmaster and see if I can get him to change the index page, and then try to reindex. Maybe there is something on the page that php dig doesn't like. Thank you for trying the url for me and quick responses. With both of us not being able to index the site leaves a lot of questions. Is it the site or something to do with phpdig. HUH??? Thank You 2-surf.net

08-25-2004, 09:45 AM	#22
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	Found Problem Here is what the problem was, Which makes no sense. The website had a robot.txt file in the that only had this line. Disallow: 4.15.191.215 I had webmaster remove the robots.txt and it indexed fine. How come the Disallow: is causing a problem with the sipder? Thank You, 2-Surf.net

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
No links found...	pwoc	Troubleshooting	0	11-10-2004 08:05 PM
links not found (well, most of them)	b2l_grefix	Troubleshooting	5	07-16-2004 11:57 AM
Please help : no links found	Danydan	Troubleshooting	2	07-12-2004 11:23 PM
Another: links found : 1	majestique	Bug Tracker	11	07-12-2004 12:19 AM
Spidering with "links found : 0"	fransdars	Troubleshooting	4	02-02-2004 12:03 AM

08-24-2004, 03:08 PM	#18
rispbiz Green Mole Join Date: Jan 2004 Posts: 15	That could be with hotmail.com but the result is the same with www.hotdial.net which does allow indexing and it is like the spider doesnt even try indexing the url. Spidering in progress... -------------------------------------------------------------------------------- SITE : http://www.hotdial.net/ Exclude paths : - - @NONE@ No link in temporary table -------------------------------------------------------------------------------- links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! If possiable could someone try indexing this site with there phpdig and let me know if they have a problem indexing it. Then I would know that if another phpdig site cant index it then it would more than likley be a problem with the url rather than an issue with my spider. If other phpdig se can index it then I would have to figure out where the problem is in my engine. Thanks for any help. 2-surf.net

08-25-2004, 04:53 PM	#24
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	I think they might have been trying to ban someone from visiting their site. Trouble is, you don't do that via robots.txt. You do it via .htaccess.