PDA

View Full Version : Long Time Spidering Crashes


mikiwiki
10-23-2006, 11:29 PM
I am spidering a huge web with a big amount of data. Very often, the web spidering (launched from admin) ends unexpectedly ("Page not found") 1, 2 or 3 hours later... Is it normal? Can a phpdig process, for example, work 24h continously?

Thanks

Charter
10-26-2006, 09:25 AM
If you are using the admin panel, and spidering via your browser, you might consider using shell instead, especially if you are spidering for hours. Not sure but it might be the case that the browser cannot handle that much text. Plus, spidering from shell should be faster.