Leith
01-21-2008, 02:16 AM
Hi Forum,
I'm trying to customize phpDig to do the following:
- I want to spider all links in a particular domain.
- Only specific URL's should be indexed, especially URL's containing the keyword - <rest of URL>"showproperty"<rest of URL>.
- My first attempt was to modify the "phpdigRewriteURL" function inside the "robot_functions.php" API, in the following way:
Code:
if (!eregi("showproperty", $eval)) {
return -1;
}
so that any URL containing the keyword is not indexed. This, however, results in all links not being followed, and only links containing the keyword...
- My second attempt involves playing with the "$ok_for_index" parameter in the "spider.php" API:
//let's go for indexing the content
if ($ok_for_index == 1) {
...
so that the "$ok_for_index" is only set to 1 if the URL being indexed contains the "showproperty" keyword.
My question is; am I going in the correct direction with this, or is there a more optimal approach?
Thanks,
Leith
I'm trying to customize phpDig to do the following:
- I want to spider all links in a particular domain.
- Only specific URL's should be indexed, especially URL's containing the keyword - <rest of URL>"showproperty"<rest of URL>.
- My first attempt was to modify the "phpdigRewriteURL" function inside the "robot_functions.php" API, in the following way:
Code:
if (!eregi("showproperty", $eval)) {
return -1;
}
so that any URL containing the keyword is not indexed. This, however, results in all links not being followed, and only links containing the keyword...
- My second attempt involves playing with the "$ok_for_index" parameter in the "spider.php" API:
//let's go for indexing the content
if ($ok_for_index == 1) {
...
so that the "$ok_for_index" is only set to 1 if the URL being indexed contains the "showproperty" keyword.
My question is; am I going in the correct direction with this, or is there a more optimal approach?
Thanks,
Leith