PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 12-30-2004, 06:26 PM   #16
jmitchell
Orange Mole
 
Join Date: Dec 2004
Location: Tennessee
Posts: 60
what if you are indexing other sites?
jmitchell is offline   Reply With Quote
Old 12-30-2004, 06:30 PM   #17
rAdoN
Green Mole
 
Join Date: Oct 2004
Posts: 27
use admin update - "Click on the noway sign to exclude from future indexings" - "Click on the cross to delete the branch" - "Click on the cross to delete a document" - that delete for links indexed not wanted - use FORBIDDEN_EXTENSIONS to prevent for sites - run the cleans - index

ps - no listen
__________________
rAdoN was here

Last edited by rAdoN; 12-30-2004 at 06:47 PM.
rAdoN is offline   Reply With Quote
Old 12-31-2004, 06:07 AM   #18
Slider
Orange Mole
 
Join Date: Jan 2004
Posts: 30
Thumbs up

Hello rAdoN,

I apoligize for being such a pain.
You really know your stuff and I will never doubt what I hear from you again.
Thank you so much for being here. Maybe I can return the favor in some way in the future.
Slider is offline   Reply With Quote
Old 12-31-2004, 01:49 PM   #19
Slider
Orange Mole
 
Join Date: Jan 2004
Posts: 30
I added this line to the config:
Code:
define('FORBIDDEN_PATH','(guestbook|forum|cgi-bin|webring|affiliates|links|webrings|banners)');
I added this code to spider.php (the part in bold red is the addition)
Code:
//test content-type of this page if not excluded
                           $result_test_http = '';
                           if (!phpdigReadRobots($exclude,$temp_path) && !eregi(FORBIDDEN_EXTENSIONS,$temp_file) && !eregi(FORBIDDEN_PATH,$temp_path)) {
                                $result_test_http = phpdigTestUrl($url_indexing,'date',$cookies);
                           }
I tried the code you gave and even tried variations of it and never was able to get it to ignore a path or directory. This code should be added to the next phpdig version. it's a neccessity if you want to have a little more control over the content that is being indexed and reduce the MySql database.
Slider is offline   Reply With Quote
Old 01-01-2005, 12:56 PM   #20
rAdoN
Green Mole
 
Join Date: Oct 2004
Posts: 27
hoorah - instead use book.cgi you make mod - good for path - i mod your mod
PHP Code:
//test content-type of this page if not excluded
$result_test_http '';
if (!
phpdigReadRobots($exclude,$temp_path.$temp_file) && !eregi(FORBIDDEN_EXTENSIONS,$temp_path.$temp_file)) {
     
$result_test_http phpdigTestUrl($url_indexing,'date',$cookies);

__________________
rAdoN was here
rAdoN is offline   Reply With Quote
Old 01-01-2005, 04:12 PM   #21
Slider
Orange Mole
 
Join Date: Jan 2004
Posts: 30
I'm not familiar with the book.cgi you were talking about.
The new code you posted would have made it work for the path and filename Congrats!

Thank you very much
__________________
Horse Search Engine
Slider is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
spidering error = theURL,winName,features ddowdall Troubleshooting 0 03-19-2006 07:28 AM
"search depth" and "links per" features laurentxav How-to Forum 1 01-12-2005 07:27 PM
Bugs, and missing Features in V. 1.6.2 Rolandks Bug Tracker 4 01-23-2004 07:01 AM
New Features Inquiry Charter Feedback & News 20 01-19-2004 07:10 PM


All times are GMT -8. The time now is 08:27 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.