PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 02-28-2005, 09:01 AM   #1
cefiro
Green Mole
 
Join Date: Feb 2005
Posts: 6
Spidering issue

I have yet to install phpdig for my website, but I would like to possibly prevent an issue that might arise once the spidering commenses. I have tried a trial of one piece of commercial software that spiders a site to create a search database like phpdig does. When I ran it I noticed that is was not spidering properly. Our site masks the true url. For example, http://www.clawfootsupply.com/product461 is a page on my site. The page that displays there a sort of 'product.php' that includes other php files to be the header and footer and other features. I think the fact that the url doesn't end in any sort of .php or.htm or something similar confused the spider. I noticed that the spider would be coming across http://www.clawfootsupply.com/product461/style68 or http://www.clawfootsupply.com/product461/type55 and others like that. There is not way from the product461 page to get to the sylte68 or type 55 pages, so why was the spider navigating that way? Will phpdig do this also? Those above "bad" links would actually take you to the product461 page, the end is ignored. You can obviously see the problems with spidering this way, since it would be indexing paths that a user would never come across on the site and be doing the same page multiple times. There is no way to get through to those links, as I mentioned, so it might have just been a bug in the spider and phpdig will not have a problem. Just thought I'd try to head off headaches form occuring later. If this is confusing let me know and I'll try to explain it a different way Thanks.
cefiro is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
pdftotext issue JonnyNoog External Binaries 6 07-14-2006 11:40 PM
config issue baskamer Troubleshooting 2 12-18-2004 12:33 PM
Installation issue... again jinx Script Installation 1 06-14-2004 08:31 PM
pstotext issue killer27 External Binaries 7 05-12-2004 01:28 PM
Spidering issue with my site pager Troubleshooting 5 01-19-2004 10:05 AM


All times are GMT -8. The time now is 12:48 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.