View Single Post
Old 11-19-2006, 08:10 AM   #4
CentaurAtlas
Green Mole
 
Join Date: Nov 2006
Location: Florida
Posts: 11
Here is some more information. The culprit is in the phpdigDetectDir routine inside robot_functions.php. For some pages it is running about 20-60 seconds when it calls
phpdigTestUrl which in turn calls fgets.

The slow part is in the fgets call, which is really strange that it is that slow on a fast connection (3 different fast connections - two on 100 mbit connections and one cable modem).

So, it is something with the network call afterall. I wrote some code in a different language that will grab a couple of pages per second.

Now I'm wondering if it is something throttling back the connection for a robot...because it should not be that slow.
CentaurAtlas is offline   Reply With Quote