View Single Post
Old 01-21-2008, 02:16 AM   #1
Leith
Green Mole
 
Join Date: Jan 2008
Posts: 1
Selective Indexing of URL Containing a <keyword>

Hi Forum,

I'm trying to customize phpDig to do the following:

- I want to spider all links in a particular domain.

- Only specific URL's should be indexed, especially URL's containing the keyword - <rest of URL>"showproperty"<rest of URL>.

- My first attempt was to modify the "phpdigRewriteURL" function inside the "robot_functions.php" API, in the following way:
Code:
if (!eregi("showproperty", $eval)) {
return -1;
}
so that any URL containing the keyword is not indexed. This, however, results in all links not being followed, and only links containing the keyword...

- My second attempt involves playing with the "$ok_for_index" parameter in the "spider.php" API:
//let's go for indexing the content
if ($ok_for_index == 1) {
...
so that the "$ok_for_index" is only set to 1 if the URL being indexed contains the "showproperty" keyword.

My question is; am I going in the correct direction with this, or is there a more optimal approach?

Thanks,
Leith
Leith is offline   Reply With Quote