I'm spidering from a shell command for the first time and having some problems. The process ran for around 5 minutes yesterday, and indexed about 50 pages, then said it was complete. I launched the shell command again, phpdig indexed a few more pages, then quit again. Same thing on a third try.
I have 1,500+ pages on my site, which indexes just fine when I run it from the secure web page, so why won't it do the same when I run it from a shell command?
