indexing doesn't work on the serverver it should

OceanSurf · 10-07-2004, 07:06 AM

Hi.

I have set up a testing phpdig on my local machine and it worked okay. When I moved the page to the 'real' server and ran the phpdig spider there to index the site it only logged the Host and nothing else. I have no idea why. chmods are the way they should be. When I indexed the site with my local machine everything worked fine. Does anyone have a clue of what is wrong?

Cheers.

vinyl-junkie · 10-07-2004, 04:58 PM

Could you post the exact error messages you're getting?

OceanSurf · 10-08-2004, 01:30 AM

it's not an error i guess, it's just that it doesn't find any links.

Spidering in progress...
SITE : http://www.the_domain.com/
Exclude paths :
- @NONE@
No link in temporary table
links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

But when I spider the same domain with my machine i get 108 links that are indexed. What's the catch? MySQL grant permissions?

dnuttall · 10-16-2004, 08:30 AM

Quote:

Originally Posted by OceanSurf

it's not an error i guess, it's just that it doesn't find any links.

Spidering in progress...
SITE : http://www.the_domain.com/
Exclude paths :
- @NONE@
No link in temporary table
links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

But when I spider the same domain with my machine i get 108 links that are indexed. What's the catch? MySQL grant permissions?

I get identical results as above.

michy · 10-28-2004, 04:47 AM

I have exact the same problem, anyone got a clue ?

dnuttall · 10-28-2004, 05:01 AM

Indexing doesn't work unless your target has a "links" via "<a href=something>something</a>" tags.

I can make it work by artificially populating the index.html/php file, but what I'd like to do is make it index what ever it finds in the directory and it seems like that means the spider has to be modified.

FWIW, I've also found that it can be somewhat unreliable in terms of stepping thru a long list of files when the files have to be converted to text. I've never been able to get thru a complete but long list of referenced files and when it quits, it leaves the system (PHPDig, not the base computer) in "locked" mode, which I have yet to figure out how to gracefully UNlock.

YMMV but so far, I don't see it as quite ready for prime time, but I would LOVE to be corrected on that! lol.

Best to all who shall hear these presents.

Dave in San Antonio, Texas.

Charter · 10-28-2004, 08:33 AM

Uh, dnuttall, then it's working as it should. PhpDig is a "web spider" and "search engine." Where does it say that PhpDig traverses directories, and what is PhpDig supposed to do if PHP is out of memory? Rhetorical questions, sure, however these "complaints" have likely been answered in one way or another somewhere on the forums.

http://www.phpdig.net/forum/showthread.php?t=1139
http://www.phpdig.net/forum/showthread.php?t=1158

instyle · 11-14-2004, 10:03 PM

I am having exactly the same issue. PhpDig works perfectly on my local machine and my local machine can spider my external domain. However when I install phpdig in my web hosting it just searches the first page BUT nothing is written to the database under keywords.

Oh, and I could see an answer to the problem in the two threads named above.

Thanks

instyle · 11-15-2004, 03:14 PM

Quote:

Originally Posted by instyle

I am having exactly the same issue. PhpDig works perfectly on my local machine and my local machine can spider my external domain. However when I install phpdig in my web hosting it just searches the first page BUT nothing is written to the database under keywords.

My problem is fixed. It was to do with permissions of the directories. had to change them to 755.

10-07-2004, 07:06 AM	#1
OceanSurf Green Mole Join Date: Sep 2004 Posts: 7	indexing doesn't work on the serverver it should Hi. I have set up a testing phpdig on my local machine and it worked okay. When I moved the page to the 'real' server and ran the phpdig spider there to index the site it only logged the Host and nothing else. I have no idea why. chmods are the way they should be. When I indexed the site with my local machine everything worked fine. Does anyone have a clue of what is wrong? Cheers.

10-08-2004, 01:30 AM	#3
OceanSurf Green Mole Join Date: Sep 2004 Posts: 7	it's not an error i guess, it's just that it doesn't find any links. Spidering in progress... SITE : http://www.the_domain.com/ Exclude paths : - @NONE@ No link in temporary table links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! But when I spider the same domain with my machine i get 108 links that are indexed. What's the catch? MySQL grant permissions? Last edited by OceanSurf; 10-08-2004 at 01:33 AM.

10-28-2004, 05:01 AM	#6
dnuttall Green Mole Join Date: Oct 2004 Posts: 7	Indexing doesn't work UNLESS... Indexing doesn't work unless your target has a "links" via "<a href=something>something</a>" tags. I can make it work by artificially populating the index.html/php file, but what I'd like to do is make it index what ever it finds in the directory and it seems like that means the spider has to be modified. FWIW, I've also found that it can be somewhat unreliable in terms of stepping thru a long list of files when the files have to be converted to text. I've never been able to get thru a complete but long list of referenced files and when it quits, it leaves the system (PHPDig, not the base computer) in "locked" mode, which I have yet to figure out how to gracefully UNlock. YMMV but so far, I don't see it as quite ready for prime time, but I would LOVE to be corrected on that! lol. Best to all who shall hear these presents. Dave in San Antonio, Texas.

10-28-2004, 08:33 AM	#7
Charter Head Mole Join Date: May 2003 Posts: 2,539	Uh, dnuttall, then it's working as it should. PhpDig is a "web spider" and "search engine." Where does it say that PhpDig traverses directories, and what is PhpDig supposed to do if PHP is out of memory? Rhetorical questions, sure, however these "complaints" have likely been answered in one way or another somewhere on the forums. http://www.phpdig.net/forum/showthread.php?t=1139 http://www.phpdig.net/forum/showthread.php?t=1158 __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

11-14-2004, 10:03 PM	#8
instyle Green Mole Join Date: Nov 2004 Posts: 3	I am having exactly the same issue. PhpDig works perfectly on my local machine and my local machine can spider my external domain. However when I install phpdig in my web hosting it just searches the first page BUT nothing is written to the database under keywords. Oh, and I could see an answer to the problem in the two threads named above. Thanks __________________ Eden Brenchley Instyle IT - Web design & Hosting

10-07-2004, 04:58 PM	#2
vinyl-junkie Purple Mole Join Date: Jan 2004 Posts: 694	Could you post the exact error messages you're getting?

10-28-2004, 04:47 AM	#5
michy Green Mole Join Date: Oct 2004 Posts: 1	I have exact the same problem, anyone got a clue ?

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
not work	isababa	Troubleshooting	3	08-30-2005 04:54 PM
PhpDig indexing won't work	sigfy	Troubleshooting	11	01-07-2005 06:47 AM
It doesn't work	humanitaire.ws	Script Installation	8	12-15-2004 03:37 AM
Indexing PDFs doesen't really work	N100101	External Binaries	3	06-18-2004 03:59 PM
Indexing word doc's OK search through files don't work	dapuse	External Binaries	3	02-05-2004 07:38 AM