Hi,
I'm trying to spider my website and need to index a lot of pages.
(easily over 120,0000) most of the spidering is done but it's just getting longer and longer to index. The site is static (a newspaper archive) and added to daily. The pages are broke down like this:
www.foo.com/years/xxxx<-(being the year)/(the issues in this format 0112 <--January 12th /
I've told phpdig to spider the site by typing out the year url - for example -
www.foo.com/years/2003/
the problems are the a) it always show's up as indexed site
www.foo.com in the control panel - not the individual years
and B) It always wants to look thru all the previous years to index
Do I have to actually create sub domains (i.e. 1996.foo.com) to have seperate directories indexed or is there some other way.
I basically want to make a static search database and don't need to reindex anything but the current days additions. Thanks in advance if you have any ideas.
Eric McClary
www.recordernews.com