PhpDig.net - View Single Post - Big dig database

JWSmythe · 05-14-2004, 10:52 PM

Thanks for the replies.

I just let it run, and it didn't seem to break anything. I really have no clue if it got all of the sites. I kinda sorta accidently killed the tasks, after like 3 days, but now the stats are:

Hosts : 3118 Entries
Pages : 31824 Entries
Index : 2421863 Entries
Keywords : 183779 Entries
Temporary table : 162490 Entries

I may change the threading scheme, so it simply monitors how many tasks are going, so I'll have a positive stopping point, rather than knowing it may have gotten to particular points in many files. Then I can put a restart into it, so if I kill it again, it can restart on the last record run, rather than starting from the top. I'll post what I do here, so if anyone's sadistic (err, interested..), they can play with it a bit, and try it themselves.

I made an interesting change, displaying thumbnails of the resulting pages, which looks rather nice, or at least I think.. I'll start another thread to put that code into.

05-14-2004, 10:52 PM	#4
JWSmythe Green Mole Join Date: May 2004 Posts: 4	Thanks. Thanks for the replies. I just let it run, and it didn't seem to break anything. I really have no clue if it got all of the sites. I kinda sorta accidently killed the tasks, after like 3 days, but now the stats are: Hosts : 3118 Entries Pages : 31824 Entries Index : 2421863 Entries Keywords : 183779 Entries Temporary table : 162490 Entries I may change the threading scheme, so it simply monitors how many tasks are going, so I'll have a positive stopping point, rather than knowing it may have gotten to particular points in many files. Then I can put a restart into it, so if I kill it again, it can restart on the last record run, rather than starting from the top. I'll post what I do here, so if anyone's sadistic (err, interested..), they can play with it a bit, and try it themselves. I made an interesting change, displaying thumbnails of the resulting pages, which looks rather nice, or at least I think.. I'll start another thread to put that code into.