command line link list not working

futureweb · 03-31-2005, 05:32 AM

Hello,

I tried to use a simple txt file to be able to have a link list that should be spidered, for some reason, that is not working. with all or forceall it works with the already known domains, but not the ones within list.txt, as I understood you use following command:
php -f [PHPDIG_DIR]/admin/spider.php list.txt

the .txt file looks like:

http://site1.com
http://site2.com
http://site3.com

but another question, is it possible to have all links within a page spidered, also of external domains, but only external domains that we specify?

like site1.com has links within site1.com but also a link to site2.com and site100.com.

I want that the links to site2.com are spidered, but not site100.com

thanks
patrick

Charter · 04-06-2005, 01:52 PM

> ...link list that should be spidered, for some reason, that is not working...

Are you using verison 1.8.7 or 1.8.8 RC1? If you pipe the output to a log file, what does the log file contain?

> ...possible to have all links within a page spidered...

Set LIMIT_TO_DIRECTORY to false in the config file.

> ...external domains, but only external domains that we specify...

Use the admin panel text box or a cron with text file.

03-31-2005, 05:32 AM	#1
futureweb Green Mole Join Date: Mar 2005 Posts: 1	command line link list not working Hello, I tried to use a simple txt file to be able to have a link list that should be spidered, for some reason, that is not working. with all or forceall it works with the already known domains, but not the ones within list.txt, as I understood you use following command: php -f [PHPDIG_DIR]/admin/spider.php list.txt the .txt file looks like: http://site1.com http://site2.com http://site3.com but another question, is it possible to have all links within a page spidered, also of external domains, but only external domains that we specify? like site1.com has links within site1.com but also a link to site2.com and site100.com. I want that the links to site2.com are spidered, but not site100.com thanks patrick

04-06-2005, 01:52 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	> ...link list that should be spidered, for some reason, that is not working... Are you using verison 1.8.7 or 1.8.8 RC1? If you pipe the output to a log file, what does the log file contain? > ...possible to have all links within a page spidered... Set LIMIT_TO_DIRECTORY to false in the config file. > ...external domains, but only external domains that we specify... Use the admin panel text box or a cron with text file. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
List and link to indexed URIs	Charter	Mod Submissions	3	01-20-2005 09:07 PM
spider from command line	twanoo	Troubleshooting	3	01-14-2005 10:04 AM
command line using winxp	mrgee	Troubleshooting	1	11-03-2004 02:20 AM
Command Line entry and nothing happens	aslan	Troubleshooting	6	08-27-2004 03:19 PM
Indexing by command line...	Canadian	How-to Forum	4	01-04-2004 06:44 PM