PhpDig.net - Error while spidering

PhpDig.net (http://www.phpdig.net/forum/index.php)

- Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)

- - Error while spidering (http://www.phpdig.net/forum/showthread.php?t=112)

jimigisme

10-01-2003 12:50 PM

Error while spidering

I've got phpdig running from the command line as follows:
C:\php>php.exe -f C:\phpdig\admin\spider.php sites.txt
THe spider runs for a bit:
Spidering in progress...
Then:
Notice: Undefined index: path in C:\phpdig\admin\robot_functions.php on line 347
Any ideas would be helpful,
Thank you!

Charter

10-01-2003 06:26 PM

What's in sites.txt, i.e., is there a full URL, one per line?

jimigisme

10-02-2003 08:27 AM

Example of sites.txt
http://www.sitename.com
http://www.sitename.com
http://www.sitename.com
http://www.sitename.com

Charter

10-05-2003 09:37 AM

Where is sites.txt located? Line 347 in robot_functions.php is:

PHP Code:


		
			
$path = $newurl["path"];

Could you change that to:

PHP Code:


		
			
$path = $newurl["path"];

echo $path . " <--- Path<br><br>";

to see what $path is?

jimigisme

10-06-2003 08:03 AM

Thank you for your help, here is what I got:

SITE : http://www.welcometoourworldofgifts.com/
Exclude paths :
- @NONE@
/welcome/default.asp <--- Path<br><br>+1:http://www.welcom
(time : 00:00:28)
/welcome/default.asp <--- Path<br><br>/welcome/default.asp

Charter

10-06-2003 02:51 PM

Hmm... How about try the following:

In the config file, set the following:

PHP Code:


		
			
define('PHPDIG_SESSID_REMOVE',true); // remove SIDS from indexed URLS

define('PHPDIG_SESSID_VAR','mscssid'); // name of the SID variable

Anyone out there ever have problems indexing *.asp files?

All times are GMT -8. The time now is 05:56 PM.