PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 06-07-2004, 03:51 PM   #1
misterbearcom
Green Mole
 
Join Date: Apr 2004
Location: Cali
Posts: 10
Rate of spidering: is it determined by the server?

I was wondering if anyone happned to know what would determine the speed of the spidering. Currently, I am spidering at an average rate of 375 URLs per hour. That seems rather slow. Would that have anything to do with the server's processor speeds? Or would it be a combination of a bunch of different factors such as:

1.) Server processor speed.
2.) Server OS.
3.) Internet bandwidth of server.
4.) My client script on my browser.

I tend to think it's 1-3 and not #4. But if anyone else has some feedback about how fast they are able to spider I would appreciate it. Thanks.
misterbearcom is offline   Reply With Quote
Old 06-07-2004, 05:48 PM   #2
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
When I spidered my site recently around noon that day, it took over 2 hours to spider about 1,500 pages. I re-spidered the same site close to midnight that same day, and it took about 45 minutes. I tend to think that during lighter traffic times on your website, the spidering process would be faster. Just a guess though.
vinyl-junkie is offline   Reply With Quote
Old 06-08-2004, 01:56 AM   #3
synnalagma
Green Mole
 
Join Date: Mar 2004
Posts: 22
1.) Server processor speed.
Of course this will change indexing speed
2.) Server OS.
Linux should be faster (it is for MySQL)
3.) Internet bandwidth of server.
If you index site that aren't on the server, of course it mather
4.) My client script on my browser.
No, this should't change anything

5) PHP configuration
If you have a low memory limit and so on it can slow indexing process
6)Load of your server
If there's ressource intensive scripts on your server this can also be scripts of your neighbours (if you're on a shared server) this can slow down indexing. Try to know where your server is located (I mean lot of european server are located in USA) to choose the right hour to do the job.
synnalagma is offline   Reply With Quote
Old 06-08-2004, 10:53 AM   #4
misterbearcom
Green Mole
 
Join Date: Apr 2004
Location: Cali
Posts: 10
Hmmm. Thanks. It makes sense.

Thanks, guys. It's very much appreciated. I had a feeling that there would be a few contributing factors. I bet my server is pretty bogged down since the number of databases being used.

In the future I suppose I will have to consider renting my own server somewhere. If anyone knows of any great rates with PHP 4.3+ and MySQL I'd appreciate it. Otherwise I was thinking about a local server company, http://www.serverbeach.com which I believe has a good rate ($99/month) for Linux Redhat. But I'm still debating about this.

Again, thanks. I really appreciate the info. I'll have to do some more brainstorming about what would be the best thing to do.
misterbearcom is offline   Reply With Quote
Old 06-08-2004, 05:44 PM   #5
vinyl-junkie
Purple Mole
 
Join Date: Jan 2004
Posts: 694
You don't say how much disk space you require. That makes a big difference in what web host anyone could recommend. My web host is MindStormHosting. I've been with them for about six months and have been very happy with their service. There are several hosting packages to choose from. You might want to check them out to see if they'd have what you need.
vinyl-junkie is offline   Reply With Quote
Old 06-08-2004, 06:20 PM   #6
misterbearcom
Green Mole
 
Join Date: Apr 2004
Location: Cali
Posts: 10
Hi Vinyl, thanks!

Quote:
Originally posted by vinyl-junkie
You don't say how much disk space you require. That makes a big difference in what web host anyone could recommend. My web host is MindStormHosting. I've been with them for about six months and have been very happy with their service. There are several hosting packages to choose from. You might want to check them out to see if they'd have what you need.
Currently, I use Neureal.com who are really great. However I know when logging on via cocoamysql that there must be at least a hundred mysql databases on the same server all running at the same time. So, it gets a bit bogged down.

I am not sure how much storage space I would need however, I am looking to grow a phpdig-based website in terms of collecting as many urls as possible but am currently on a limited budget, so I really do not know as of yet. However, more of anything in terms of hardware and software would always be better, me thinks.
misterbearcom is offline   Reply With Quote
Old 06-09-2004, 05:39 AM   #7
robertDouglass
Green Mole
 
Join Date: Jun 2004
Posts: 3
Spidering own server: optimization?

I was wondering if there are any optimizations one can make when spidering a site hosted on the same server (same domain)? In particular, if I tell phpdig to spider www.mydomain.com, doesn't this involve the DNS server and a roundtrip to the internet? I tried localhost, but that didn't work (shared hosting). Any suggestions?
__________________
-Rob
------
visit me at www.robshouse.net
robertDouglass is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Spider indexing/request speed/rate - How to change? JAB Creations How-to Forum 1 09-07-2005 05:46 AM
Moving to a new server ezytrak How-to Forum 1 03-03-2005 08:11 AM
Hello, I use a Windows Server ClausBrell The Mole Hole 2 09-30-2004 04:35 AM
Test Server RaGe Mod Requests 0 05-10-2004 04:01 PM
Spidering Problems on a Windows Server Website vinyl-junkie Troubleshooting 23 02-20-2004 06:44 PM


All times are GMT -8. The time now is 09:36 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.