|
11-01-2004, 10:26 AM | #1 | |
Green Mole
Join Date: Jul 2004
Posts: 2
|
Fix for slow spidering in PhpDig 1.8.x
Some people using PhpDig reported severe perfomance impact when moving from version 1.6.x to 1.8.x.
I was one of them. It could take more the 20 seconds for me to index a single page with version 1.8.x. PhpDig 1.6.x indexed the same page in less then a second. This happened only under win32. Finally I found the cause of it. Look at the following code: PHP Code:
As stated in HTTP/1.1 http://www.w3.org/Protocols/rfc2616/rfc2616-sec14.html Quote:
To fix this search for the following in robot_functions.php file: ."User-Agent: PhpDig/".PHPDIG_VERSION." (+http://www.phpdig.net/robot.php)".END_OF_LINE_MARKER.END_OF_LINE_MARKER; There will be a total of 3 matches (lines 354, 460 and 628) Insert before each of them this line: ."Connection: close".END_OF_LINE_MARKER Hope this will help. Last edited by vital; 11-01-2004 at 10:54 AM. |
|
11-02-2004, 12:20 AM | #2 |
Orange Mole
Join Date: Nov 2003
Posts: 42
|
This correction will make a big difference for spidering speed. I suggest that everybody (and Charter) will implement this immediately! Thank you very much.
-m- |
11-03-2004, 11:48 AM | #3 |
Green Mole
Join Date: Oct 2004
Posts: 5
|
Wow, I think that fix really helped me. I applied it and it looks good!
Great work! -Brian Sutton http://www.piedmontswingdance.org/search |
11-06-2004, 11:33 AM | #4 |
Green Mole
Join Date: Sep 2004
Posts: 25
|
I tried it, but I am not seeing any faster results. According to my log file, pages at my site are still being indexed at around 60 - 90 seconds per page.
|
Thread Tools | |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
Problem in PhpDig Spidering | navdeep.madan | Troubleshooting | 0 | 10-04-2006 04:51 AM |
phpdig blocked when spidering any site | heli | Troubleshooting | 3 | 09-30-2004 11:42 AM |
Indexing slow.... no, _really_ slow | bluntman | Troubleshooting | 1 | 09-24-2004 02:23 PM |
speciffically slow spidering at fgets() | slintz | Troubleshooting | 7 | 08-18-2004 03:24 AM |
Spidering **VERY** Slow | Niall Fernie | Troubleshooting | 4 | 07-13-2004 01:45 AM |