Thread: Slow indexing
View Single Post
Old 02-27-2008, 05:30 AM   #1
Alex-FBTech
Green Mole
 
Join Date: Feb 2008
Posts: 1
Slow indexing

Hello!

I just want Phpdig to index around 40 000 pages, and I saw that other people made this without problems. I cannot give the root of my website in the admin panel because it's only a search engine and without a list of links, it's not possible to crawl the entire site.

At the beginning I made an html file with all the links of my site, and I launched the spider on it with a 1 depth level but it seemed to had some problems to catch too much links in a page.

I tried to divide my links in 10 files of around 4000 links.
Now, Phpdig has no problems to just catch quickly the links in the temporary table, but it takes around 30/40 seconds per page when it begin to index! (~ 10 days to index the site)

I tried with the web interface, with shell commands and I had the same results. I set always with 1 depth level and 0 links per.

How can I do to speed up the spider?

Last edited by Alex-FBTech; 02-27-2008 at 05:34 AM.
Alex-FBTech is offline   Reply With Quote