Command line spider ignoring "filelist.txt"
 

I can run a command line spider by doing the following:

>> php - f search.php
The above indexes everything...

However - I want to just update a few sites, so I have a file called "filelist.txt" which contains 2 URLS's (each on a diffent line)...

When I do the following:

>>php - f search.php filelist.txt
It makes no difference... phpdig proceeds to update everything!!!!

Any ideaS?...
I need to resolve this so i can setup a cron....

Also - FYI:
>> php -f search.php http://www.anydomain.com
This works as expected... It's just with specifying a filelist that there is problems.

Thanks in advance
LH

Hi. What version of PhpDig are you using?

1.6.x (as far as i can see - taken from "spider.php" file)

Update - I upgraded to latest version, and STILL it ignores the filelist.txt and simply begins indexing everything!!!

Please help!!
LH

Update:
Have tried and tried with this, but no luck.

I've opted to just run 6 crons (instead of 1), in the format:
php - f path/to/dir/spider.php http://theurliwanttoindex.com

Not pretty, but it works....
Still open to any suggestions if you have any...

LH

Hi. Try updating to 1.8.3.

Charter - I did upgrade - and still the same result....
:(

Is the tempspider table empty?

That seems to have done the trick!!!! :)
Does the Temporary table always need to be empty then?...
Only I see that it's often got entries in it...

LH

Hi. The tempspider table should try and empty after PhpDig is done, but if it doesn't (for example, the spidering process prematurely ends) then just go and empty that table.