[tibabs]

Hi,

Since 2 days I try to dig 20.000 html pages on my database that already contains close to 6.000.000 records in phpdig_engine table.
Unfortunately ... it last a very very long time (it could last more than 1 minute to dig a unique html file as it took around 3 seconds 2 weeks ago !!!).
After several investigation concerning my system (XP) my database limits (Mysql), my folder size (20.000 html files) ... I've found the solution.
I hope it could help somebody else.

Using time-tracker function I've discover that the time consuming code is the "Optimizing phase" of the spider.php file (PhpDig V1.8.0). As a result ... just comment this 4 lines, integrate another optimizing process your own way (every 5000 digs for example) and enjoy with your new boosted Phpdig.

=== Code to comment in spider.php
//print "Optimizing tables...".$br;
//@mysql_query("OPTIMIZE TABLE ".PHPDIG_DB_PREFIX."spider",$id_connect);
@mysql_query("OPTIMIZE TABLE ".PHPDIG_DB_PREFIX."engine",$id_connect);
@mysql_query("OPTIMIZE TABLE ".PHPDIG_DB_PREFIX."keywords",$id_connect);


Remarks: I'm only using PhpDig for inserting new html files. There is no update, no delete. By the way the original PhpDig optimization phase is less important for me.

Regards.
tibabs.