Here is some more information. The culprit is in the phpdigDetectDir routine inside robot_functions.php. For some pages it is running about 20-60 seconds when it calls
phpdigTestUrl which in turn calls fgets.
The slow part is in the fgets call, which is really strange that it is that slow on a fast connection (3 different fast connections - two on 100 mbit connections and one cable modem).
So, it is something with the network call afterall. I wrote some code in a different language that will grab a couple of pages per second.
Now I'm wondering if it is something throttling back the connection for a robot...because it should not be that slow.
|