View Single Post
Old 02-28-2005, 12:55 PM   #1
nevsie
Green Mole
 
Join Date: Feb 2005
Posts: 3
Spidering Stops, Site is Locked after 7 Pages

Hi,

I am using PHPDIG and like the programme... However, i have a problem with the spidering stopping after a set amount of pages. One site it might be 7 pages another it might be 10, but never more. The site then stays in the state of being locked.

I have set:
Depth to '10'
Links per '0'
'LIMIT_TO_DIRECTORY',false
'PHPDIG_IN_DOMAIN',true

Other comments have been that you should use a browser such as FireFox to stop timeout issues... I am using Firefox, although i never see the second page where the details of page spidering arise... Does this mean the browser is timing out... If so can i change the timeout period, as i cannot access shell to run PHP DIG.

Alternatively could my host be ending the spidering. Whether it be at the mysql end or at the spidering end? I have contacted them and asked, but as of yet no reply.

Any help is truely appreciated.

Thanks N
nevsie is offline   Reply With Quote