Spidering Stops, Site is Locked after 7 Pages
 

Hi,

I am using PHPDIG and like the programme... However, i have a problem with the spidering stopping after a set amount of pages. One site it might be 7 pages another it might be 10, but never more. The site then stays in the state of being locked.

I have set:
Depth to '10'
Links per '0'
'LIMIT_TO_DIRECTORY',false
'PHPDIG_IN_DOMAIN',true

Other comments have been that you should use a browser such as FireFox to stop timeout issues... I am using Firefox, although i never see the second page where the details of page spidering arise... Does this mean the browser is timing out... If so can i change the timeout period, as i cannot access shell to run PHP DIG.

Alternatively could my host be ending the spidering. Whether it be at the mysql end or at the spidering end? I have contacted them and asked, but as of yet no reply.

Any help is truely appreciated.

Thanks N

Checked with the hosting company and they have nothing on the mysql or scan prevention side that would stop this.

Therefore i can only assume it is to do with the browser timing out. If this is the case, what can i do about it. I have tried in various browsers from mozilla, firefox, IE and netscape...

Any help is appreciated

Hi Nevsie,
Just to understand : you have installed phpdig on your website and you run it with your browser trhough the admin panel (something like www.yoursite.com/phpdig/admin/) ?
Your hosting company say that they don't block (nor timeout) the execution of the script so you think it comes from the browser ?
...
Maybe Charter can help you...

Is it possible for the browser to "timeout" the loading of a page ?

I don't know...
sorry,
Julien.

Hi All,

As much as my hosts admitted it was not them doing anything... I was contacted by one of their higher techies...

He loved the script, and had not seen it before!!!

But he informed me that they have a maximum script execution time!!! So they did stop the script after all!!! Thats the answer, so cheers guys...

The solution is to scan groups, sub groups, or single pages at a time. Then it works great.

Cheers, N