Index some, but spider all pages
Hi all,
Have a Question:
The index page of the domain i want to spider, contains links to say level1.php?id=xx and level1.php?id=yy
level1.php?id=xx contains links to level2.php?id=aa and level2.php?id=bb
level1.php?id=yy contains links to level2.php?id=cc and level2.php?id=dd
The pages i want to index are the level2.php pages. But i only can reach them trough index -> level1.php -> level2.php.
So the spider needs to follow all the links from the root.
What would be the best way to index some, but spider all pages?
|