Go Back > PhpDig Forums > Troubleshooting

Thread Tools
Old 09-23-2004, 01:40 PM   #1
Green Mole
Join Date: Sep 2004
Posts: 2
Indexing slow.... no, _really_ slow

Hi all,

I'm spidering a Windows 2000 server based site with several thousand pages on it. The indexing seems to be going really slowly. I don't mean 3-4 seconds, as most everyone else seems to be calling slow. It is taking over 1 minute for each page to index.

The results are the same whether I run the spider from the Windows 2000 host or an external Linux machine. The Linux machine does seem to run better in other respects (spidering doesn't stop from the web interface, etc), but it still takes the same amount of time.

Does anyone have any idea how to troubleshoot this?

bluntman is offline   Reply With Quote
Old 09-24-2004, 02:23 PM   #2
Wayne McBryde
Orange Mole
Join Date: Oct 2003
Location: NC, USA
Posts: 34
If you post the URL of the site, I will try spidering it with PHPDig versions 1.6.3 & 1.8.3 and post back the time per page to spider. Of course I don't plan to spider the several thousand pages you got but I will spider 10 to 20 pages to get an average.

I currently spider over 1000 sites for my database and some sites do spider slow like the page per minute you talked about. Others will spider faster in the range of 3 to 4 seconds per page. I always guessed the slow sites were slow because of the server they were running on.
Wayne Mcbryde
We search all of Lake Norman!
Wayne McBryde is offline   Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow indexing Alex-FBTech Troubleshooting 0 02-27-2008 06:30 AM
Indexing are very slow, how can i speed up? ossi01 Troubleshooting 2 05-06-2005 04:26 AM
helpme indexing slow xorex Troubleshooting 3 07-27-2004 07:00 AM
Indexing very slow mdkn Troubleshooting 2 03-21-2004 02:01 PM
Very Slow Indexing airplay Troubleshooting 2 03-09-2004 03:20 PM

All times are GMT -8. The time now is 02:46 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.