PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)
-   -   not spidering all pages (too many links on page?) (http://www.phpdig.net/forum/showthread.php?t=1258)

mirdin 09-01-2004 04:38 AM

not spidering all pages (too many links on page?)
 
I have trouble getting all my pages spidered.

I don't think it has to do with levels, since it does find some pages at deeper levels. However it doesn't spider all links on pages.

I was thinking it might have to do with the LINKS_MAX_LIMIT being 20.

In config.php I've tried changing LINKS_MAX_LIMIT to 100, but that doesn't seem to make any difference.

Any idea?

BulForce 09-01-2004 06:00 AM

mirdin, i think that if you leave the box "links per:" at 0 this mean for spider
to follow all the links in the page.

mirdin 09-01-2004 06:08 AM

That's what I thought also, I did leave it at 0 a couple of times and that didn't work.

After trying a few times with limit=10 en perlinks=100 it worked, so no more problems here :)


All times are GMT -8. The time now is 02:51 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.