PHP spidering nuisance

Psycho1 · 04-17-2004, 09:52 AM

I just installed PHPDig on my site today and I'm impressed! It followed my robots.txt file so it didn't crawl my forums.....but I have a mod for phpBB that adds posts from a forum on my homepage....and it has several pages of posts on my homepage with urls like mysite.com/index.php?start=5&sid=afdhjdsahfksdahkflafsadkhjfasfs

It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process?

Thanks in advance for any help.

Psycho1 · 04-17-2004, 10:45 AM

I guess I'm retarded because I can't figure out how to stop the spidering process. Do I just click delete site from the admin panel...because I haven't found anything else. I don't want to hit delete site and lose data though.

Psycho1 · 04-17-2004, 10:50 AM

Doh, don't tell me all I had to do was hit stop in my browser

misterbearcom · 04-17-2004, 08:35 PM

Hi, I'm new here, but I believe in the spidering page you can always go to the address bar, cut out the spider.php file name to bring you back to the admin/ which will default back to your index.php page. At that point the site you were indexing will probably be locked up.

At that point go to the list of domains, select the domain you just spidered (even if it says it's not locked treat it like it was), then click on the update button and then on the new window for that domain there will be a button that says the site is locked and give you an option to unlock it. Afterwards go back to admin and start doing whatever it is you want to do...

Hope this helps. (I'm sure there's an easier way in case the Stop button in the toolbar is the better solution but it works for me...) Cheers.

04-17-2004, 09:52 AM	#1
Psycho1 Green Mole Join Date: Apr 2004 Posts: 9	PHP spidering nuisance I just installed PHPDig on my site today and I'm impressed! It followed my robots.txt file so it didn't crawl my forums.....but I have a mod for phpBB that adds posts from a forum on my homepage....and it has several pages of posts on my homepage with urls like mysite.com/index.php?start=5&sid=afdhjdsahfksdahkflafsadkhjfasfs It likes to visit these types of links many, many times before it moves on to other parts of my site. Is it possible for PHPDig to be setup to only crawl php pages that do not have variables in the url? And how would I go about stopping the spidering so I can fix this issue or can I fix this without stopping the spidering process? Thanks in advance for any help.

04-17-2004, 10:45 AM	#2
Psycho1 Green Mole Join Date: Apr 2004 Posts: 9	I guess I'm retarded because I can't figure out how to stop the spidering process. Do I just click delete site from the admin panel...because I haven't found anything else. I don't want to hit delete site and lose data though.

04-17-2004, 10:50 AM	#3
Psycho1 Green Mole Join Date: Apr 2004 Posts: 9	Doh, don't tell me all I had to do was hit stop in my browser

04-17-2004, 08:35 PM	#4
misterbearcom Green Mole Join Date: Apr 2004 Location: Cali Posts: 10	Hi, I'm new here, but I believe in the spidering page you can always go to the address bar, cut out the spider.php file name to bring you back to the admin/ which will default back to your index.php page. At that point the site you were indexing will probably be locked up. At that point go to the list of domains, select the domain you just spidered (even if it says it's not locked treat it like it was), then click on the update button and then on the new window for that domain there will be a button that says the site is locked and give you an option to unlock it. Afterwards go back to admin and start doing whatever it is you want to do... Hope this helps. (I'm sure there's an easier way in case the Stop button in the toolbar is the better solution but it works for me...) Cheers.