cannot spider own site - others yes
 

Im sure that its something quite simple.

I have been able to successfully spider certain sites and currently show the following stats.

DataBase status
Hosts 140
Pages 7622
Index 2007006
Keywords 170156
Excluded paths 101
Temporary table 143915
Log entries 643



However, I still cannot seem to spider our own root site.
This seems top have been a problem for others posting here as well. Is it a domain name problem? Im not sure. I have complete control over my dns and it work correctly but still cannot spider our root site

A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped.

Any help would be appreciated

__________________
http://www.thewebnewsroom.com

actually just figured it out. pretty simple too.

Seems that if I delete the robots.txt it will allow me to spider.

So ill go back and delete the directories I dont want indexed and then replace the tobots.txt

thanks