Not Spidering Deep enough (strange site i have tho)

[bigals]

Hi i went over this problem in the thread I psted below, after I managed to get my site to spider my php files, i came accroos the problem of not spidering deep enough:

Old Thread:
http://www.phpdig.net/showthread.php?s=&threadid=764&perpage=15&pagenumber=2

It spiders 20 deep, and thats as far as it goes, but I have folders that contain over 60 php files, and each one needs to be spidered, so can i set the spider depth to about 70?

so for these big folders, 20 of the php fiiles are spidered and the rest are not...

can someone explain how to dig a little deeper :)

cheers,

Alex.

[vinyl-junkie]

This is just a guess, but have you tried changing this code in config.php:define('SPIDER_MAX_LIMIT',20);
to this?define('SPIDER_MAX_LIMIT',70);

[bigals]

O right I see, i'll try that and see how it goes!

cheers!

[bigals]

DAM its not working!

Its still spidering only 20 results, i think its because my php page are only generating 20 results for the page, when it reaches 20 results it creates a button that links to the next 20, the button will refresh the page and list results 20 - 40,

you see all the links that are sopidered are created by php and don't exist anywhere else, is that why its not spidering anymore than 20 do you think?

it sounds like its right.

:)

[Charter]

Hi. PhpDig follows links. If there are no links to the other pages, then PhpDig won't find them.

[bigals]

Yeah cool!

I did what we said before, i've made 1 php page that generates a huge list of links to every index page and it works but it seems to miss some of them,

i only spidered 1 deep because every link was in the same documnet, perhaps if a spider a little deeper it might find the ones its missing out?

alex.

[asanad]

Hi all

I am trying to crawl my website, for some reason PHPDig is only crawling very partially my website, it does not go deaper than the homepage.

Best Regards:
Amjad Sanad

[vinyl-junkie]

What search depth and links per values did you choose? If you left them both at the default of zero, you'll only get the home page crawled. Try changing the search depth to 5, for example, and see what happens. Depending on how many pages phpdig spiders you'll have to set that value even higher.

Hope this helps. :)

[asanad]

I set the depth to 20 and links to 0, but, still it is not spidering deep enough
only certain documents in the homepage.

[vinyl-junkie]

Find the following in config.php:define('SPIDER_MAX_LIMIT',20);
and change it to this:define('SPIDER_MAX_LIMIT',40);
Re-spider and see what you get.