View Full Version : Spider Directories

10-09-2003, 12:43 PM
I just installed PhpDig...am impressed with its search speed, but have one problem. It spidered only two of my subdirectories, and when I try to spider these, I see the message that this domain was spidered recently.

http://mydomain.com - root - was spidered
http://mydomain.com/directory1&2 were spidered....

but http://mydomain.com/directory3,4 etc. were not spidered. How do I accomplish this?

Thanks for your time.

10-09-2003, 04:46 PM
Hi. Are there links from mydomain.com to directories three and four? Do you get the message that the site was recently indexed? If so, you might try changing define('LIMIT_DAYS',7); to define('LIMIT_DAYS',1); in the config file.

10-09-2003, 05:17 PM
Hi Charter,

Thanks for your prompt response.

Yes, I'm getting the message the site was recently indexed, so I changed define('LIMIT_DAYS',7); to define('LIMIT_DAYS',1); in the config file....still same result...site was recently indexed.

But you bring up an interesting point...many of these subdirectories are linked from the home page, but only through a dhtml/javascript menu navigation. Perhaps that is part of my problem?

10-09-2003, 05:23 PM
Ah, Javascript. Perhaps this thread (http://www.phpdig.net/showthread.php?threadid=86) might help.

Also, if you want to try reindexing from scratch, you might find this thread (http://www.phpdig.net/showthread.php?threadid=78) useful.

You could also try setting define('LIMIT_DAYS',1); to define('LIMIT_DAYS',0);

10-09-2003, 06:18 PM
Thanks again for the help.

I was able to spider the remaining directories, except for one that does not have an html link. I could just create the link temporarily if it is that important.

Thanks for the good utility! - it's nice to be off of the web-based, outsourced search. :)