PhpDig.net - View Single Post

hendrix · 02-10-2005, 01:23 PM

Hi,

I've had a problem indexing a particular site (please note that all other sites have been indexed without any problem).

PhpDig v1.8.7 is located at http://www.santeestrie.qc.ca/recherche

I've tried to index http://www.iugs.ca but it always returned a 404 error. So then I tried indexing a file I knew existed (http://www.iugs.ca/FR/100/RH_Recrutement.asp) but it also returned a 404 error:
------------------------------------------------
HTTP/1.1 404 Object Not Found - http://www.iugs.ca/FR/100/RH_Recrutement.asp
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
404s are either dead links or something looked like a link to PhpDig so PhpDig tried to crawl it.
Optimizing tables...
Indexation terminée !
------------------------------------------------

It doesn't matter which page I try to index on this site, it will never work. There's no robot.txt so that's not the problem.

Here are a few of my settings:

- Tried indexing with a depth of 10 and links per set to zero.

define('PHPDIG_IN_DOMAIN',true);
define('SPIDER_MAX_LIMIT',20);
define('RESPIDER_LIMIT',5);
define('LINKS_MAX_LIMIT',20);
define('RELINKS_LIMIT',5);
define('LIMIT_TO_DIRECTORY',false);
define('LIMIT_DAYS',0);

and from phpinfo():

allow_url_fopen = 1
safe_mode = off

Any help would be appreciated

Regards,
Stéphane Brault
eComDEV.com

02-10-2005, 01:23 PM	#1
hendrix Green Mole Join Date: Feb 2005 Posts: 10	404 error although page exists Hi, I've had a problem indexing a particular site (please note that all other sites have been indexed without any problem). PhpDig v1.8.7 is located at http://www.santeestrie.qc.ca/recherche I've tried to index http://www.iugs.ca but it always returned a 404 error. So then I tried indexing a file I knew existed (http://www.iugs.ca/FR/100/RH_Recrutement.asp) but it also returned a 404 error: ------------------------------------------------ HTTP/1.1 404 Object Not Found - http://www.iugs.ca/FR/100/RH_Recrutement.asp See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation. 404s are either dead links or something looked like a link to PhpDig so PhpDig tried to crawl it. Optimizing tables... Indexation terminée ! ------------------------------------------------ It doesn't matter which page I try to index on this site, it will never work. There's no robot.txt so that's not the problem. Here are a few of my settings: - Tried indexing with a depth of 10 and links per set to zero. define('PHPDIG_IN_DOMAIN',true); define('SPIDER_MAX_LIMIT',20); define('RESPIDER_LIMIT',5); define('LINKS_MAX_LIMIT',20); define('RELINKS_LIMIT',5); define('LIMIT_TO_DIRECTORY',false); define('LIMIT_DAYS',0); and from phpinfo(): allow_url_fopen = 1 safe_mode = off Any help would be appreciated Regards, Stéphane Brault eComDEV.com Last edited by hendrix; 02-10-2005 at 01:26 PM.