View Full Version : Problem spidering sites at in .txt over 20 address

01-30-2004, 03:55 AM
I have trouble spidering sites from a text file containing websites that has over a certain amount of address.

when I enter the following command with 300 sites in the file sites.txt

php -f spider.php sites.txt

I get
spidering in progress...

When I check only 50 websites are added as HOST and in the TEMP tables but not as entires.

When I spider the with only 20 sites in the file sites.txt it works fine.

how can I place huge amounts of sites in a .txt file and have all spidered

01-30-2004, 11:05 AM
Hi. Do the URLs start with http and are they listed one per line in the text file? If so, how long do you wait before you check the tables?

01-30-2004, 08:07 PM
Yes I have the address begin with http:// .....and one line for each address.

for 300 address I wait about 20 second then I check the tables

01-30-2004, 08:13 PM
Hi. Not sure of the exact time frame with 300 sites, but try giving it a couple of hours, checking the tables intermittently, but don't stop PhpDig while checking.