Quote:
Originally posted by vinyl-junkie
I am able to spider your site, albeit very slowly. I suspect that's because of all your robots exclusions (my gosh, there's a lot!)
Are you sure you have your write permissions setup correctly? Check the program documentation on that. It's most likely what is tripping you up.
BTW, welcome to the forum! 
|
I wrote a few proggies to help me maintain the site. One small part of that creates a robots.txt file so that I wouldn't have to keep going through the site and try to figure out where I added new image folders, etc. So yeah, it's kind of big, but a click of a button to maintain.
problem is I can't get phpdig to work on my site locally. I'm running apache server on my desktop and having the same problem. I just don't get it. ???