PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 11-25-2005, 11:54 AM   #1
dewed
Green Mole
 
Join Date: Nov 2005
Posts: 1
Question Can I make the spider stop and start on a dime?

I have a specific 36 hour window every week I'm allowed to spider a remote 300k+ page catalog site. I had been using wget in recursive mode, but I have no good way to stop it and restart where it left off the next week. I also have my own format I'd like the data stored in the mysql table.

Can phpdig be bent to meet my needs? or am I better off writing my own using curl and a very large database of urls to crawl, driven by a bash/perl/php script frontend?
dewed is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
How To stop spider by shell command ? noel How-to Forum 4 11-03-2005 01:06 PM
Only searching from start of word benklocek Troubleshooting 1 03-18-2005 01:14 PM
How can I make phpdig spider faster jakeres How-to Forum 1 11-29-2004 11:05 AM
Fixing spider.php, protecting from locking site after timeout or users stop Konstantine Mod Submissions 3 04-09-2004 12:37 PM
where do i start for installing this script ekimbo Script Installation 1 03-24-2004 10:41 PM


All times are GMT -8. The time now is 09:03 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.