PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 04-20-2005, 06:10 AM   #1
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
Lightbulb PhpDig only with Mysql

I want to make Phpdig work only with Mysql because the folder text_content take much disk space on my server (100MB) , so I want to know if somebody have already made this ??????


excuse me my english is not perfect
vodevil is offline   Reply With Quote
Old 04-20-2005, 08:37 AM   #2
bloodjelly
Purple Mole
 
Join Date: Dec 2003
Posts: 106
Making PhpDig store the content in SQL tables won't take up any less space (I guess unless you used compression). You can limit the storage of text content by changing this line in the config file:

Code:
define('CONTENT_TEXT',0);
This way no text files will be created. Otherwise I guess you'd have to make some fulltext tables, then go through the code, and everywhere you see an fopen (in phpdigWriteText (robot_functions.php) for example), change it to an SQL query.
bloodjelly is offline   Reply With Quote
Old 04-20-2005, 04:02 PM   #3
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
Ok bloodjelly at this time a test this solution , if it work i'll submit my modified phpDig version to the community !
vodevil is offline   Reply With Quote
Old 04-21-2005, 03:28 PM   #4
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
Angry

I have modifieed this function :


Quote:
//=================================================
//store a content_text from a spider_id
function phpdigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT == 1) {

$file_text_path = $relative_script_path.'/'.TEXT_CONTENT_PATH.$spider_id.'.txt';
if ($f_handler = @fopen($file_text_path,'w')) {
reset($text);
while (list($n_chunk,$text_to_store) = each($text)) {
fputs($f_handler,wordwrap($text_to_store));
}
fclose($f_handler);
@chmod($file_text_path,0666);
//here the ftp case
if (FTP_ENABLE) {
$ftp_id = phpdigFtpKeepAlive($ftp_id);
@ftp_delete($ftp_id,$spider_id.'.txt');
$res_ftp = false;
$try_count = 0;
while (!$res_ftp && $try_count++ < 10) {
$res_ftp = @ftp_put($ftp_id,$spider_id.'.txt',$file_text_path,FTP_ASCII);
if (!$res_ftp) {
sleep(2);
}
}
if (!$res_ftp) {
print "Ftp_put error !".$br;
}

}
}
else {
print "Warning : Unable to create the content file $file_text_path ! $br";
}
}
return $ftp_id;
}

By this :

Quote:
//=================================================
//store a content_text from a spider_id
function phpdigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT == 1) {


while (list($n_chunk,$text_to_store) = each($text)) {
fputs($f_handler,wordwrap($text_to_store));
}

$requete = "INSERT INTO text_content VALUES ('$spider_id','$text_to_store')";


$query = mysql_query($requete) or die(mysql_error);

}


}
This modification no cause error but phpdig doesn't store information into Mysql
vodevil is offline   Reply With Quote
Old 04-21-2005, 08:02 PM   #5
AllKnightAccess
Green Mole
 
Join Date: Sep 2004
Posts: 25
I would also like a way to store the text into MySQL, instead of the server. Why is it stored on the server in the first place? Wouldn't the content be better placed in the database?
AllKnightAccess is offline   Reply With Quote
Old 04-22-2005, 09:40 AM   #6
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
You don't necessarily have to modify the PhpDig 1.8.7 code to store text in MySQL. Try setting the following in the config file instead:
Code:
define('SUMMARY_DISPLAY_LENGTH',150);            //Max chars displayed in summary

define('DISPLAY_SNIPPETS',false);                 //Display text snippets

define('DISPLAY_SUMMARY',true);                 //Display description

define('CONTENT_TEXT',0);                    //Activates/deactivates the
                                             //storage of text content.
In search_function.php of PhpDig 1.8.7, here is some relevant code:
Code:
             if (DISPLAY_SUMMARY) {
                 $table_results[$n]['text'] = htmlspecialchars(phpdigHighlight($reg_strings,preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",ereg_replace('(@@@.*)','',wordwrap($text, SUMMARY_DISPLAY_LENGTH, '@@@')))),ENT_QUOTES);
                 $table_results[$n]['text'] = phpdigSpanReplace($table_results[$n]['text']);
             }
In robot_functions.php of PhpDig 1.8.7, here is some relevant code:
Code:
$db_some_text = preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",implode("",$text));
if (strlen($db_some_text) > SUMMARY_DISPLAY_LENGTH) {
  $db_some_text = substr($db_some_text,0,SUMMARY_DISPLAY_LENGTH)."...";
}

$first_words = preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",$titre_resume)."\n".preg_replace("/([ ]{2,}|\n|\r|\r\n)/"," ",$page_desc['content'].$db_some_text)."...";
Note that up to SUMMARY_DISPLAY_LENGTH characters will get displayed in the search results, and that the highlighting of keywords works differently than when storing text in files, so make sure to try a demo before going live.

This was all changed/improved in PhpDig 1.8.8 RC1 so that highlighting can work the same whether or not you store text in files, and PhpDig 1.8.8 RC1 has a TEXT_STORAGE_AMOUNT option so that you can set how much text to store.

If using PhpDig 1.8.8 RC1, you'll probably want the following config options set to store text in MySQL but not files:
Code:
define('DISPLAY_SNIPPETS',true);                 // display text snippets

define('DISPLAY_SUMMARY',false);                 // display description

define('SNIPPET_DISPLAY_LENGTH',150);            // max chars displayed in each snippet

define('TEXT_STORAGE_AMOUNT',10000);             // max characters per page to store in files/tables

define('CONTENT_TEXT',0);                        // activates/deactivates the storage of text content in files
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 04-23-2005, 07:58 AM   #7
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
My solution don't work , it's the good way but phpdig storing nothing into Mysql .
vodevil is offline   Reply With Quote
Old 04-30-2005, 05:11 PM   #8
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
This is my new modification

Code:
//=================================================
//delete a spider reccord and content file
function PhpDigDelSpiderRow($id_connect,$spider_id,$ftp_id='')
{
global $relative_script_path,$ftp_id;
$query = "DELETE FROM ".PhpDig_DB_PREFIX."engine WHERE spider_id=$spider_id";
$result_id = mysql_query($query,$id_connect);
$query = "DELETE FROM ".PhpDig_DB_PREFIX."spider WHERE spider_id=$spider_id;";
$result_id = mysql_query($query,$id_connect);
// PhpDigDelText($relative_script_path,$spider_id,$ftp_id);
   PhpDigDelText($spider_id,$ftp_id);
}



// =============================
// Ecriture dans Mysql
// print "Erreur d'écriture dans Mysql ! $br";

function PhpDigWriteText($relative_script_path,$spider_id,$text,$ftp_id='') {
global $br;
if (CONTENT_TEXT ==1) {

$file_text_path = mysql(" SELECT * FROM 'PhpDigtext_content' ORDER BY 'spider_id' ");

if($f_handler = $file_text_path ) {
reset($text);
while (list($n_chunk,$text_to_store) = each($text)) {

mysql_query(" INSERT INTO PhpDigtext_content (spider_id, content) VALUES ('$spider_id','$text_to_store')", $id_connect);
  }

}

// Here the mysql case
if (FTP_ENABLE) {
$ftp_id = PhpDigFtpKeepAlive($ftp_id);

// @delete @ftp_delete($ftp_id,$spider_id.'.txt');

$delete = mysql_query(" DELETE 'PhpDigtext_content' WHERE 'spider_id' = '$spider_id' ");
$res_ftp = false;
$try_count = 0;
while (!$res_ftp && $try_count++ < 10) {

  $res_ftp = mysql(" INSERT INTO PhpDigtext_content (spider_id) VALUES ('$spider_id'), $id_connect);
                if (!$res_ftp) {
                            sleep(2);

}

                  if (!$res_ftp) {

                                   echo 'Unable to wgite in ! $br';

         }
 
      }

    }

       else {
              echo 'Warning : Imposible d\'écrire dans Mysql file $file_text_path ! $br';

 }

}

 return $ftp_if;

}

//=================================================
//delete a content_text from a spider_id
function PhpDigDelText($relative_script_path,$spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{

//there delete the ftp file
if (FTP_ENABLE && $ftp_id)
             @$delete = 1;
}
}



// =============================

function PhpDigFtpConnect()
{
 if (CONTENT_TEXT == 1 && FTP_ENABLE== 1) {
 $count = 0;
 global $br;

 while ($count++ < 10) {

                $ftp_id = 1;


    }
    
    return $ftp_id; 
 
  }
   sleep(2); 
 
}


// =============================
// close ftp if exis

 function PhpDigFtpClose($ftp_id)
{
 if ($ftp_id)
    @ftp_quit($ftp_id);
}

It's work at 98% but this function , cause a bug ,


Code:
//=================================================
//delete a content_text from a spider_id
function phpdigDelText($relative_script_path,$spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{


$query = "SELECT * FROM 'phpdigtext_content' WHERE spider_id = '$spider_id' ";

$file_text_path =mysql_query($query,$id_connect);


//there delete the ftp file


if (FTP_ENABLE && $ftp_id)
    $delete = mysql_query(" DELETE FROM 'phpdigtext_content' WHERE spider_id=$spider_id ");
}

}
vodevil is offline   Reply With Quote
Old 04-30-2005, 05:16 PM   #9
vodevil
Green Mole
 
Join Date: Mar 2005
Posts: 11
Code:
//=================================================
//delete a content_text from a spider_id
function xunDelText($spider_id,$ftp_id='')
{
if (CONTENT_TEXT == 1)
{

if (FTP_ENABLE && $ftp_id)
    $delete = mysql_query(" DELETE * FROM 'xuntext_content' WHERE spider_id=$spider_id ");


   }

}

Don't work
vodevil is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Using non-root MySQL account with phpDig muppet How-to Forum 0 01-22-2006 07:08 AM
PhpDig and Mysql without text_content vodevil Script Installation 1 04-06-2005 01:42 PM
PHPDig 1.8.8 RC1 and MySQL djavet Script Installation 2 01-31-2005 11:02 PM
Create mysql user for phpdig motopsycho How-to Forum 0 05-10-2004 06:11 AM
PHPDig or MySQL Full Text ? Korben How-to Forum 4 03-24-2004 10:59 PM


All times are GMT -8. The time now is 01:35 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.