![]() |
|
![]() |
#1 |
Green Mole
Join Date: Feb 2005
Posts: 1
|
![]()
Hi,
I'm new to phpdig. The installation was quite easy, though it was not quite clear to me that cookies have to be enabled for administration. It was fairly easy to create a nice template, but at this point I'm stuck, so I come for help. When I tried to really index my site, the spider silently ignored a lot of pages. The description of 1.8.8-rc1 states that the spider does not index directory listings and database content. And by looking at some php code I found out that it does some extensive checking on other things as well. I did some debugging and found out that indeed the spider doesn't like most of my urls, even though I have added a space, both parenthes and other characters to the $allowed_link_chars. But in my case, I really need to index all content. Without that ability phpdig would be of no use. Is there a setting that I can use, or some code I can add or comment out in order to make the spider more greedy? I've had a look at the code of the spider but it's complex with many nested statements, so I'm reluctant to change it. I figured some of you could perhaps answer my question from the top of their heads. Maarten Wijnen |
![]() |
![]() |
![]() |
#2 |
Head Mole
Join Date: May 2003
Posts: 2,539
|
Set "search depth" to a large number, set "links per" to zero, choose the "no" option, set LIMIT_TO_DIRECTORY to false, set PHPDIG_IN_DOMAIN to true.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension. |
![]() |
![]() |
![]() |
#3 |
Green Mole
Join Date: Mar 2005
Posts: 1
|
Well Maarten, I have today installed PhpDig and found the same problem as you. It does index dynamic content but what i noticed is that it will index ALL information when the query is ON the page with the variables hardcoded. If you dynamically insert the variables it doesn't seam to pass the variables. I have about 4 pages to which variables are sent as opposed to hardcoded into the query. I told PhpDig to index each of these pages manually and all is well.
However my setup is that ALL the variables are in my menus and the same menu appears on EVERY page, including the dynamically created catalogue pages in my case. Maybe that's why my workaround works. Peter |
![]() |
![]() |
![]() |
|
|
![]() |
||||
Thread | Thread Starter | Forum | Replies | Last Post |
specific links not spider | flanders | How-to Forum | 1 | 10-06-2004 11:12 PM |
Spider site with links | dell_10 | External Binaries | 10 | 09-20-2004 06:42 AM |
no spider my file links | lolodev | Troubleshooting | 21 | 07-16-2004 06:31 PM |
Index ignores directory with space in it | Gakk Simian | Troubleshooting | 2 | 04-09-2004 09:03 AM |
phpDig ignores robots.txt | Dragonfly | Troubleshooting | 1 | 09-12-2003 06:54 AM |