I had the problem phpdigExplore() returns to many duplicate links. This caused the spider to check 100s of duplicate URLs, which caused a slowdown, and the 1000 pages limit was hit quite fast.
Finally I added the following code at the end of phpDigExplore():
PHP Code:
if(!$_SESSION["links"]) $_SESSION["links"]=array();
$resultlinks = array();
foreach($links as $link){
if(!array_search($link, $_SESSION["links"])){
$_SESSION["links"][]=$link;
$resultlinks[]=$link;
}
}
return $resultlinks;
I don't know whether this modification is useful or harms other components. But for the moment, it works.