Hi Charter, I wonder if you can shed a little light on an indexing problem I have found?
A friend has asked me to index his website for him and I have tried, each time the spider goes off to the host but doesn't find any content. No errors come back at all. Or are displayed.
This is the first time this has happened and I started to wonder about it, the spider is still working on other domains really well with no problems.
The website in question can be found at
www.adhd.org.nz/Auckland1.html which appears to be just another site.
I then looked up the hosting provider and then tried to index another few websites that are hosted with them and the results were the same for the other domains as well.
The spider just bounced off.
I can't imagine the problem is caused by a robots.txt setting which excludes the user agent and I am told by the webmaster that no robots.txt file is in place?
So I was wondering if it may relate to a HTACCESS file or other settings that the host may have, to stop some user agents visiting and indexing.
So have you any ideas about this at all? Perhaps you have a way of testing to see if the agent is blocked for some reason.
Any suggestions would be good, I am a little perplexed by this.
Isn't it stunning to realise that it is already March 2006 where is the year going to??
Heaps of regards Charter & everyone else who is using the forum.