PhpDig.net - View Single Post - Indexing cookie/session authenticated pages

bforsyth · 06-27-2004, 08:41 AM

Thanks for the welcome. Testing for the user agent doesn't seem to be working (although it is hard to tell as I don't currently have access to the logs.....

Is there any other way that I could tell if a script is being called by PHPDig?

Also, when the spider does a crawl, it seems to dismiss dynamically generated pages that differ only in ids:

eg: ?page=articleView&articleId=250
giving Duplicate of an existing document

There are several thousand of these articles and it looks like none of them are being indexed....

06-27-2004, 08:41 AM	#8
bforsyth Green Mole Join Date: Jun 2004 Posts: 22	Thanks for the welcome. Testing for the user agent doesn't seem to be working (although it is hard to tell as I don't currently have access to the logs..... Is there any other way that I could tell if a script is being called by PHPDig? Also, when the spider does a crawl, it seems to dismiss dynamically generated pages that differ only in ids: eg: ?page=articleView&articleId=250 giving Duplicate of an existing document There are several thousand of these articles and it looks like none of them are being indexed....