PDA

View Full Version : Problem spidering sites at in .txt over 20 address


joshuag200
01-30-2004, 04:55 AM
I have trouble spidering sites from a text file containing websites that has over a certain amount of address.

when I enter the following command with 300 sites in the file sites.txt

php -f spider.php sites.txt

I get
spidering in progress...

When I check only 50 websites are added as HOST and in the TEMP tables but not as entires.

When I spider the with only 20 sites in the file sites.txt it works fine.

how can I place huge amounts of sites in a .txt file and have all spidered

Charter
01-30-2004, 12:05 PM
Hi. Do the URLs start with http and are they listed one per line in the text file? If so, how long do you wait before you check the tables?

joshuag200
01-30-2004, 09:07 PM
Yes I have the address begin with http:// .....and one line for each address.

for 300 address I wait about 20 second then I check the tables

Charter
01-30-2004, 09:13 PM
Hi. Not sure of the exact time frame with 300 sites, but try giving it a couple of hours, checking the tables intermittently, but don't stop PhpDig while checking.