Go Back > PhpDig Forums > Troubleshooting

Thread Tools
Old 03-09-2004, 02:48 PM   #1
Green Mole
Join Date: Mar 2004
Posts: 7
Very Slow Indexing

Hi, this is my first attempt at using php dig.
I have a site that has around 32,000 pages.
When I first ran the spider it ran overnight and had around 10,000 pages indexed when I woke up.
I notced there was a long time inbetween pages (around 6 seconds). I found the "sleep" in your code so I went ahead and changed it to 2 seconds.

I then cleared out all of the tables and started over.
This sped up the process at first, but now it has been 24 hours and the darn thing is still running.
Is this normal for it to take this long to index 32,000 pages?

Any ideas?

airplay is offline   Reply With Quote
Old 03-09-2004, 03:18 PM   #2
Head Mole
Charter's Avatar
Join Date: May 2003
Posts: 2,539
Hi airplay, and welcome to!

With many pages, perhaps set the following in the config.php file, where X is one or two:
PHP Code:
define('LIMIT_DAYS',0);              //default days before reindex a page
define('SPIDER_MAX_LIMIT',X);        //max recurse levels in spider
define('SPIDER_DEFAULT_LIMIT',X);    //default value
define('RESPIDER_LIMIT',X);          //recurse limit for update 
and then crawl your site in chunks.

One thing I've noticed is that users in general tend to set the search depth to the highest possible value and then let the robot run. This tends to get a lot of repeat documents, lending to a longer index time.

Also, when you want to start over, it might be better to delete the site from the admin panel, as this will empty the tables (execpt for keywords and logs) and delete the TXT files. The clean dictionary link will clean/empty the keywords table, but it is probably faster to do it from shell, and the logs tables would need to be emptied from shell or phpMyAdmin.
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 03-09-2004, 03:20 PM   #3
Green Mole
Join Date: Mar 2004
Posts: 7
Excellent! Thanks for the quick reply! I'll give that a try and let you know how it goes!

airplay is offline   Reply With Quote

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Slow indexing Alex-FBTech Troubleshooting 0 02-27-2008 06:30 AM
Indexing are very slow, how can i speed up? ossi01 Troubleshooting 2 05-06-2005 04:26 AM
Indexing slow.... no, _really_ slow bluntman Troubleshooting 1 09-24-2004 02:23 PM
helpme indexing slow xorex Troubleshooting 3 07-27-2004 07:00 AM
Indexing very slow mdkn Troubleshooting 2 03-21-2004 02:01 PM

All times are GMT -8. The time now is 03:37 AM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2020, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.