PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 10-23-2006, 11:29 PM   #1
mikiwiki
Green Mole
 
Join Date: Apr 2006
Posts: 4
Long Time Spidering Crashes

I am spidering a huge web with a big amount of data. Very often, the web spidering (launched from admin) ends unexpectedly ("Page not found") 1, 2 or 3 hours later... Is it normal? Can a phpdig process, for example, work 24h continously?

Thanks
mikiwiki is offline   Reply With Quote
Old 10-26-2006, 09:25 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
If you are using the admin panel, and spidering via your browser, you might consider using shell instead, especially if you are spidering for hours. Not sure but it might be the case that the browser cannot handle that much text. Plus, spidering from shell should be faster.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Long Call help. GunMuse How-to Forum 0 07-26-2006 10:11 AM
Long post, possible issues, and maybe the data to solve some? Paul D. Buck Troubleshooting 4 02-03-2005 11:59 AM
How long? sjSJ How-to Forum 1 05-11-2004 06:09 PM
PDFs in directory listed as one long entry motopsycho External Binaries 4 03-09-2004 05:07 PM
Request Time mongeon Mod Requests 1 10-05-2003 10:15 AM


All times are GMT -8. The time now is 03:54 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2019, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.