Problem spidering sites at in .txt over 20 address
 

I have trouble spidering sites from a text file containing websites that has over a certain amount of address.

when I enter the following command with 300 sites in the file sites.txt

php -f spider.php sites.txt

I get
spidering in progress...

When I check only 50 websites are added as HOST and in the TEMP tables but not as entires.

When I spider the with only 20 sites in the file sites.txt it works fine.

how can I place huge amounts of sites in a .txt file and have all spidered

Hi. Do the URLs start with http and are they listed one per line in the text file? If so, how long do you wait before you check the tables?

Yes I have the address begin with http:// .....and one line for each address.

for 300 address I wait about 20 second then I check the tables

Hi. Not sure of the exact time frame with 300 sites, but try giving it a couple of hours, checking the tables intermittently, but don't stop PhpDig while checking.