View Full Version : Spider process does not do anything

04-27-2004, 09:58 AM
I'm encountering a problem getting the spider to work. Creating a robots.txt file did not seem to help.

Here is my robots.txt:

User-agent: *
Disallow: /include/

When I tell it to spider localhost or the hostname of the local server (depth=5) always results in this:

Spidering in progress...

SITE : http://localhost/
Exclude paths :
- include/

There are no CPU cycles being used and the page is fully loaded. What else can I check? How will I know that the spider process is actually doing something?


05-01-2004, 07:32 AM
Hi. Try this (http://www.phpdig.net/showthread.php?threadid=514) thread.

05-10-2004, 09:26 AM
I took a look at the article and I don't think that is the problem. I have a single web server with one NIC. I'm trying to spider the local web server, but I never see any results. I don't see any progress on the page (which also says it is "Done") and it will sit there forever doing nothing. I tried spidering some other sites but nothing happens with those either.

What else is there that I can check?

05-12-2004, 02:33 AM
In config.php set up define('USE_IS_EXECUTABLE_COMMAND','0');

(It is the best search script!)

05-12-2004, 04:48 AM
That did the trick!!! The spider page didn't load, but if I refresh the admin page I can see the counts going up. And I can actual search for things on the site. Thanks for your help Siava!