PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   Troubleshooting (http://www.phpdig.net/forum/forumdisplay.php?f=22)
-   -   Slow indexing (http://www.phpdig.net/forum/showthread.php?t=2854)

Alex-FBTech 02-27-2008 05:30 AM

Slow indexing
 
Hello!

I just want Phpdig to index around 40 000 pages, and I saw that other people made this without problems. I cannot give the root of my website in the admin panel because it's only a search engine and without a list of links, it's not possible to crawl the entire site.

At the beginning I made an html file with all the links of my site, and I launched the spider on it with a 1 depth level but it seemed to had some problems to catch too much links in a page.

I tried to divide my links in 10 files of around 4000 links.
Now, Phpdig has no problems to just catch quickly the links in the temporary table, but it takes around 30/40 seconds per page when it begin to index! (~ 10 days to index the site)

I tried with the web interface, with shell commands and I had the same results. I set always with 1 depth level and 0 links per.

How can I do to speed up the spider?


All times are GMT -8. The time now is 01:08 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.