PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 09-01-2004, 04:38 AM   #1
mirdin
Green Mole
 
Join Date: Apr 2004
Posts: 14
not spidering all pages (too many links on page?)

I have trouble getting all my pages spidered.

I don't think it has to do with levels, since it does find some pages at deeper levels. However it doesn't spider all links on pages.

I was thinking it might have to do with the LINKS_MAX_LIMIT being 20.

In config.php I've tried changing LINKS_MAX_LIMIT to 100, but that doesn't seem to make any difference.

Any idea?
mirdin is offline   Reply With Quote
Old 09-01-2004, 06:00 AM   #2
BulForce
Orange Mole
 
Join Date: Aug 2004
Location: none
Posts: 33
mirdin, i think that if you leave the box "links per:" at 0 this mean for spider
to follow all the links in the page.
BulForce is offline   Reply With Quote
Old 09-01-2004, 06:08 AM   #3
mirdin
Green Mole
 
Join Date: Apr 2004
Posts: 14
That's what I thought also, I did leave it at 0 a couple of times and that didn't work.

After trying a few times with limit=10 en perlinks=100 it worked, so no more problems here
mirdin is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Not spidering mod rewritten pages wineo Troubleshooting 1 02-22-2006 07:53 PM
Spider indexes cgi pages but not its links!? WebSpider Troubleshooting 3 02-08-2005 06:04 PM
Spidering stops after 6 level 1 pages Steve Troubleshooting 3 05-20-2004 04:56 PM
Restrict Spidering to new pages only bushpilot How-to Forum 1 05-01-2004 08:07 AM
Speed up spidering by skipping internal page links peterpeter Mod Requests 6 04-18-2004 07:17 AM


All times are GMT -8. The time now is 03:52 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.