Long Time Spidering Crashes
 

I am spidering a huge web with a big amount of data. Very often, the web spidering (launched from admin) ends unexpectedly ("Page not found") 1, 2 or 3 hours later... Is it normal? Can a phpdig process, for example, work 24h continously?

Thanks

If you are using the admin panel, and spidering via your browser, you might consider using shell instead, especially if you are spidering for hours. Not sure but it might be the case that the browser cannot handle that much text. Plus, spidering from shell should be faster.