Psycho1
04-17-2004, 09:52 AM
I just installed PHPDig on my site today and I'm impressed! It followed my robots.txt file so it didn't crawl my forums.....but I have a mod for phpBB that adds posts from a forum on my homepage....and it has several pages of posts on my homepage with urls like mysite.com/index.php?start=5&sid=afdhjdsahfksdahkflafsadkhjfasfs
It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process?
Thanks in advance for any help.
It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process?
Thanks in advance for any help.