PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 05-14-2004, 01:40 PM   #1
Steve
Green Mole
 
Join Date: May 2004
Posts: 2
Spidering stops after 6 level 1 pages

Hi

Having got this this brilliant search engine to nearly work, I'm having trouble getting it to go all the way...

No matter which page I start the spider on, it appears to always stop after it's spidered 6 level 1 pages. Site remains 'locked' (easily unlocked) and I'm clearing out the database before each attempt, just to make sure, but the problem seems persistent.

I've read all the other relevant posts that I can find, but can't see an answer to this problem (though I don't think I'm the only user to see it).

I've made only one change in config.php - set limit days to 0.

Is there a definitive answer to this problem - something basic in the setup maybe?

Thanks in advance for helping me out ....

Steve
Steve is offline   Reply With Quote
Old 05-14-2004, 04:51 PM   #2
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
Hi, Steve, and welcome to the forum!

I assume you've made sure the folder permissions are set correctly. What site are you trying to spider? And how are you trying to do it? Through the admin panel, or a cron job? Are you getting some sort of error, or is the spider just hanging? Answers to these questions might give us a start on how to help you.
vinyl-junkie is offline   Reply With Quote
Old 05-20-2004, 01:38 PM   #3
Steve
Green Mole
 
Join Date: May 2004
Posts: 2
Hi vinyl-junkie

Just to keep you updated......

I set up phpdig 1.8.0 on my own server and tried to spider the remote site and it worked perfectly - no spider hang ups, so this is obviously a clue! Config was exactly the same. I guess this is something to do with my ISP server - maybe it's got some sort of time limit on running php scripts (?). Anyway, I guess it's not your problem, since I've proved that the spidering works properly - it's just a pain in the arse when when it bombs out on my ISP's server. Can I solve this with a cron job? (I've heard of these somewhere, but I don't have a clue what they are!

Thanks for a really great software package!

Steve
Steve is offline   Reply With Quote
Old 05-20-2004, 04:56 PM   #4
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
Quote:
Originally posted by Steve
Can I solve this with a cron job? (I've heard of these somewhere, but I don't have a clue what they are!
Glad to hear you were able to spider your site. We like success stories around here.

There is an excellent tutorial on cron jobs here. Basically it's like a batch job that runs on whatever dates and times you specify, so you can spider your site automatically.

Hope this helps.
vinyl-junkie is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Spidering always stops after 5-6 minutes digdug Troubleshooting 1 10-04-2006 11:17 AM
Spider stops before all pages are indexed halide Troubleshooting 3 07-19-2005 12:26 AM
Indexing stops after a few pages Sibona Troubleshooting 1 05-03-2005 10:27 AM
Spidering Stops, Site is Locked after 7 Pages nevsie Troubleshooting 3 03-02-2005 03:52 PM
spidering stops after 21 miniutes fritze71 Troubleshooting 1 01-18-2005 05:11 AM


All times are GMT -8. The time now is 04:12 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.