PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 05-23-2004, 03:28 PM   #1
web newsroom
Green Mole
 
Join Date: May 2004
Posts: 4
cannot spider own site - others yes

Im sure that its something quite simple.

I have been able to successfully spider certain sites and currently show the following stats.

DataBase status
Hosts 140
Pages 7622
Index 2007006
Keywords 170156
Excluded paths 101
Temporary table 143915
Log entries 643



However, I still cannot seem to spider our own root site.
This seems top have been a problem for others posting here as well. Is it a domain name problem? Im not sure. I have complete control over my dns and it work correctly but still cannot spider our root site

A qualified subdomain.mydamain.com will work? I have changed the robots.txt and the .htaccess file and still stumped.

Any help would be appreciated

__________________
http://www.thewebnewsroom.com
__________________
http://www.thewebnewsroom.com
web newsroom is offline   Reply With Quote
Old 05-23-2004, 03:32 PM   #2
web newsroom
Green Mole
 
Join Date: May 2004
Posts: 4
actually just figured it out. pretty simple too.

Seems that if I delete the robots.txt it will allow me to spider.

So ill go back and delete the directories I dont want indexed and then replace the tobots.txt

thanks
web newsroom is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Imcomplete spider : locked site but nothing append beausej Troubleshooting 1 07-18-2005 03:48 PM
Spider site with links dell_10 External Binaries 10 09-20-2004 06:42 AM
spider only one site/file jdc32 Troubleshooting 2 07-02-2004 05:49 AM
Help Please: Problems to spider... just one site 1001studio Troubleshooting 1 06-28-2004 04:41 PM
spider entire site summitmn Troubleshooting 1 05-03-2004 04:30 PM


All times are GMT -8. The time now is 10:21 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.