PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 02-17-2005, 12:45 PM   #1
josegringo
Green Mole
 
Join Date: Feb 2005
Posts: 2
Question New Exclude Option

greetings!

I am trying to accomplish the spidering of a site which has an event calendar that I would like to index. This site is not my own, so the usual methods for excluding links do not apply.

My question is this: how hard would it be to add a "do not follow" rule for links? The site I am spidering has a series of links which follow this pattern:

index.htm?cYear=2002
index.htm?cYear=2003
index.htm?cYear=2004
index.htm?cYear=2005

In the interest of not spending a ton of time indexing things that I don't want, and to minimize the intrusion on their site (I am using their bandwidth), I would like to have a rule, perhaps in the config file that basically says: if cYear is not = 2005 then don't follow the link.

Any thoughts on this would be appreciated. I am ok with PHP, but not good enough to dig into the code and find where to hard code the exception statement. Once I knew where the statement went, I could code it though...

Thanks,
-josegringo
josegringo is offline   Reply With Quote
Old 02-17-2005, 01:16 PM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
config file...

FORBIDDEN_EXTENSIONS in PhpDig < 1.8.8
FORBIDDEN in PhpDig 1.8.8+

http://www.phpdig.net/forum/showthread.php?t=1684
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 02-17-2005, 02:48 PM   #3
josegringo
Green Mole
 
Join Date: Feb 2005
Posts: 2
Thumbs up Thanks

Charter,

Thanks for quickly setting me in the right direction. I spent a good part of the day running in circles. I guess I just wasn't searching for the right term in the forum.

Cheers,
-Joey
josegringo is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
default option should be to subscribe to threads you've created? rwillmer Feedback & News 0 08-27-2005 03:28 AM
option=any dont work Puller Troubleshooting 0 12-22-2004 03:19 PM
Alternate Page Option When Indexing vinyl-junkie Mod Requests 0 12-12-2004 10:12 AM
Exclude paths : -'*' -@NONE@ BootsWalker Troubleshooting 2 10-20-2004 06:12 PM
"limit to" option on from page? atticus1214 How-to Forum 3 03-03-2004 01:05 PM


All times are GMT -8. The time now is 07:17 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.