Log in

View Full Version : Quick continue after accident and regular spiderind time.


sf44
07-07-2004, 02:17 AM
Hi,
How to quick continue spidering/indexing after break?
My computer have crashed during spiderind/indexing proces. I am using Mozilla browser.
The site is locked - I m able to unlock it in admin, but what next ?
Update takes a lot of time, I just want to continue.
I am trying to spider more then 2000 pages ~ 30kB - I have download them to local computer.
But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer.
Is it regular ?

Slavomir Slovakia.

sf44
07-07-2004, 09:03 AM
I have solved this problem. Is is nescessary to work with database.
After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based.

Slavomir.

Charter
07-07-2004, 03:59 PM
Hi. Here are some answers, although you may have already figured them out. :)

>> How to quick continue spidering/indexing after break?

If spidering ends prematurely, there may be index info remaining in the tempspider table. If so, just enter the URL again or choose the site and update.

>> The site is locked - I m able to unlock it in admin, but what next ?

See above.

>> But spidering sites takes too long - about 400 pages - 24 hours and it takes longer and longer.
Is it regular ?

Maybe update to version 1.8.1 or make sure that the tempspider table is empty between runs. To empty the tempspider table, just click the delete button without selecting a site.

>> Is is nescessary to work with database.

Some people using Win/IIS servers have had to fiddle with the database tables, but if you are using Apache, then it shouldn't be necessary to work directly with the tables.

>> After some testing I have found that at my computer script which run from console is about 10 times more quick then browser based.

Welcome to shell. :D