Can I make the spider stop and start on a dime?
 

I have a specific 36 hour window every week I'm allowed to spider a remote 300k+ page catalog site. I had been using wget in recursive mode, but I have no good way to stop it and restart where it left off the next week. I also have my own format I'd like the data stored in the mysql table.

Can phpdig be bent to meet my needs? or am I better off writing my own using curl and a very large database of urls to crawl, driven by a bash/perl/php script frontend?