|
Download -
PHP Reference -
Links -
Demo 1.8.9 RC1 -
API
License - Credits - Copying - Changelog - Documentation |
|
|
09-01-2004, 04:38 AM
|
#1 |
|
Green Mole
Join Date: Apr 2004
Posts: 14
|
not spidering all pages (too many links on page?)
I have trouble getting all my pages spidered.
I don't think it has to do with levels, since it does find some pages at deeper levels. However it doesn't spider all links on pages. I was thinking it might have to do with the LINKS_MAX_LIMIT being 20. In config.php I've tried changing LINKS_MAX_LIMIT to 100, but that doesn't seem to make any difference. Any idea? |
|
|
|
09-01-2004, 06:00 AM
|
#2 |
|
Orange Mole
Join Date: Aug 2004
Location: none
Posts: 33
|
mirdin, i think that if you leave the box "links per:" at 0 this mean for spider
to follow all the links in the page. |
|
|
|
|
09-01-2004, 06:08 AM
|
#3 |
|
Green Mole
Join Date: Apr 2004
Posts: 14
|
That's what I thought also, I did leave it at 0 a couple of times and that didn't work.
After trying a few times with limit=10 en perlinks=100 it worked, so no more problems here 
|
|
|
|
 |
|
|
Similar Threads
|
||||
| Thread | Thread Starter | Forum | Replies | Last Post |
| Not spidering mod rewritten pages | wineo | Troubleshooting | 1 | 02-22-2006 07:53 PM |
| Spider indexes cgi pages but not its links!? | WebSpider | Troubleshooting | 3 | 02-08-2005 06:04 PM |
| Spidering stops after 6 level 1 pages | Steve | Troubleshooting | 3 | 05-20-2004 04:56 PM |
| Restrict Spidering to new pages only | bushpilot | How-to Forum | 1 | 05-01-2004 08:07 AM |
| Speed up spidering by skipping internal page links | peterpeter | Mod Requests | 6 | 04-18-2004 07:17 AM |
