Index some, but spider all pages
Hi all,
Have a Question: The index page of the domain i want to spider, contains links to say level1.php?id=xx and level1.php?id=yy level1.php?id=xx contains links to level2.php?id=aa and level2.php?id=bb level1.php?id=yy contains links to level2.php?id=cc and level2.php?id=dd The pages i want to index are the level2.php pages. But i only can reach them trough index -> level1.php -> level2.php. So the spider needs to follow all the links from the root. What would be the best way to index some, but spider all pages? |
All times are GMT -8. The time now is 02:51 AM. |
Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.