PhpDig.net - View Single Post - Problem spidering sites at in .txt over 20 address

joshuag200 · 01-30-2004, 03:55 AM

I have trouble spidering sites from a text file containing websites that has over a certain amount of address.

when I enter the following command with 300 sites in the file sites.txt

php -f spider.php sites.txt

I get
spidering in progress...

When I check only 50 websites are added as HOST and in the TEMP tables but not as entires.

When I spider the with only 20 sites in the file sites.txt it works fine.

how can I place huge amounts of sites in a .txt file and have all spidered

01-30-2004, 03:55 AM	#1
joshuag200 Green Mole Join Date: Jan 2004 Posts: 10	Problem spidering sites at in .txt over 20 address I have trouble spidering sites from a text file containing websites that has over a certain amount of address. when I enter the following command with 300 sites in the file sites.txt php -f spider.php sites.txt I get spidering in progress... When I check only 50 websites are added as HOST and in the TEMP tables but not as entires. When I spider the with only 20 sites in the file sites.txt it works fine. how can I place huge amounts of sites in a .txt file and have all spidered __________________ Best Regards, Joshua