Go Back > PhpDig Forums > Troubleshooting

Thread Tools
Old 09-01-2004, 05:38 AM   #1
Green Mole
Join Date: Apr 2004
Posts: 14
not spidering all pages (too many links on page?)

I have trouble getting all my pages spidered.

I don't think it has to do with levels, since it does find some pages at deeper levels. However it doesn't spider all links on pages.

I was thinking it might have to do with the LINKS_MAX_LIMIT being 20.

In config.php I've tried changing LINKS_MAX_LIMIT to 100, but that doesn't seem to make any difference.

Any idea?
mirdin is offline   Reply With Quote
Old 09-01-2004, 07:00 AM   #2
Orange Mole
Join Date: Aug 2004
Location: none
Posts: 33
mirdin, i think that if you leave the box "links per:" at 0 this mean for spider
to follow all the links in the page.
BulForce is offline   Reply With Quote
Old 09-01-2004, 07:08 AM   #3
Green Mole
Join Date: Apr 2004
Posts: 14
That's what I thought also, I did leave it at 0 a couple of times and that didn't work.

After trying a few times with limit=10 en perlinks=100 it worked, so no more problems here
mirdin is offline   Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Not spidering mod rewritten pages wineo Troubleshooting 1 02-22-2006 08:53 PM
Spider indexes cgi pages but not its links!? WebSpider Troubleshooting 3 02-08-2005 07:04 PM
Spidering stops after 6 level 1 pages Steve Troubleshooting 3 05-20-2004 05:56 PM
Restrict Spidering to new pages only bushpilot How-to Forum 1 05-01-2004 09:07 AM
Speed up spidering by skipping internal page links peterpeter Mod Requests 6 04-18-2004 08:17 AM

All times are GMT -8. The time now is 11:51 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.