PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 01-13-2004, 05:41 AM   #1
haxored
Green Mole
 
Join Date: Jan 2004
Posts: 1
spidering links but not their text

I've got a bunch of pages with a navigation menu on it, as well as some text I don't want indexed.

I want to exclude everything but a <div> tag that contains all the text I want to index.. However...

it seems that if I do that, all of the links in my navigation menu aren't spidered. That is not helpful.

How can I get phpDig to spider my entire site, but only index one section of every page?
haxored is offline   Reply With Quote
Old 01-14-2004, 09:32 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. You might try using PHPDIG_EXCLUDE_COMMENT and PHPDIG_INCLUDE_COMMENT from the config file, each on their own line, to exclude a portion of a page. Depending on your navigation menu, it might not be getting indexed because PhpDig excludes certain tags from index. If this is the case, you might try setting up a simple HTML page with links to your site, and then crawl this page. Once the crawl is done, you can delete the simple page from the admin panel.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 01-27-2004, 01:23 AM   #3
MaXius
Green Mole
 
Join Date: Jan 2004
Location: Melbourne, Australia
Posts: 2
I have a similar problem,

I have a pulldown menu, (div tags) and phpdig pulls all the menu options into the search results, hence most words you would search for are on alll pages. in a big ugly mess.

Does phpdig still crawl the links within a PHPDIG_EXCLUDE_COMMENT area even tho it doesnt put these into its database?

Thanks
MaXius is offline   Reply With Quote
Old 01-27-2004, 08:58 AM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. The exclude/include comments are for omitting parts of a page from indexing.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 01-27-2004, 11:49 AM   #5
MaXius
Green Mole
 
Join Date: Jan 2004
Location: Melbourne, Australia
Posts: 2
Yeh, gathered that... does it still spider the links within an excluded part tho...?

Ta
MaXius is offline   Reply With Quote
Old 01-27-2004, 04:01 PM   #6
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. Using the below simple example, PhpDig works as follows:
Code:
<html>
<body>
This text is indexed
<!-- phpdigExclude -->
<a href="http://www.this_link.com/is_followed.html">This text is ignored</a>
This text is also ignored
<!-- phpdigInclude -->
</body>
</html>
To change this behavior, the phpdigExplore and/or phpdigIndexFile functions in robot_functions.php could be modified.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Change phpdigExclude to not spider Links as well as text gkdeveloper How-to Forum 0 09-12-2007 03:54 AM
Exclude links from indexing, keep text digger_123 How-to Forum 0 12-20-2006 04:14 AM
Text ads and sponsored links? WebSpider Mod Requests 2 03-09-2005 12:17 AM
Text Links Showing up in Keywords List DannyD How-to Forum 2 09-24-2004 10:02 PM
Spidering....links found : 0 -IAN- Troubleshooting 23 08-25-2004 04:53 PM


All times are GMT -8. The time now is 06:59 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.