Hi -
I'm using the hack I made to limit number of links spidered (
this post) so this might be a problem of my making and if it is, sorry.
Anyway when I try to index a URI through a text file, PhpDig spiders it, limiting the links to the number I set, but then goes on to spider the rest of the already indexed URIs in the DB without the limit. Right now, pDig is seems to be in a loop, spidering wetcanvas.com (
logfile:http://www.tjhdesign.com/spider.log) This makes me think it is most definitely because of an error with my code changes on the post listed at the top.
Regardless of my code changes though, should pDig re-spider URIs outside of the ones given to it by the text file? Any idea on why it's looping? The limit code works very well when spidering through a browser, it just seems to have problems when using a text file. Thanks
