PhpDig.net - View Single Post - Site not updating

guinessec · 11-30-2004, 01:53 PM

Hi,

When I try an spider a site again from the admin control panel, no new links are found (even though I know there are some there).
The site did successfuly index a portion of the site initially, but form this point on, no further pages were indexed then, or on any further spider attempt.

Output:
SITE : http://www.mydomain.com/
Exclude paths :
- @NONE@
<SNIP>
No link in temporary table
links found : 0

My 'LIMIT_DAYS' is set to 0 and my php allow_url_fopen is set to On, safemode is off, and I am trying to respider by clicking on the 'green checkmark' next to the 'root' in the update section of the phpdig contol panel.

I have also tried a fresh attempt by using these instructions from another thread:

# empty all the PhpDig database tables
# delete all files that may be in the temp dir
# delete all files in the text_content dir except keepalive.txt
# run spider.php from a browser

When I try to telnet, as suggested in an older thread, I do recieve a "Connection closed by foreign host." which seems to suggest that 'head requests' may be a problem???? Er, maybe?

So my question, how does the server admin allow head request, and are there any security implications?

My site has over 35000 pages, but I'm sure that isn't an issue for phpdig.

Thanks,

Dale.

11-30-2004, 01:53 PM	#1
guinessec Green Mole Join Date: Nov 2004 Posts: 3	Site not updating - head requests? Hi, When I try an spider a site again from the admin control panel, no new links are found (even though I know there are some there). The site did successfuly index a portion of the site initially, but form this point on, no further pages were indexed then, or on any further spider attempt. Output: SITE : http://www.mydomain.com/ Exclude paths : - @NONE@ <SNIP> No link in temporary table links found : 0 My 'LIMIT_DAYS' is set to 0 and my php allow_url_fopen is set to On, safemode is off, and I am trying to respider by clicking on the 'green checkmark' next to the 'root' in the update section of the phpdig contol panel. I have also tried a fresh attempt by using these instructions from another thread: # empty all the PhpDig database tables # delete all files that may be in the temp dir # delete all files in the text_content dir except keepalive.txt # run spider.php from a browser When I try to telnet, as suggested in an older thread, I do recieve a "Connection closed by foreign host." which seems to suggest that 'head requests' may be a problem???? Er, maybe? So my question, how does the server admin allow head request, and are there any security implications? My site has over 35000 pages, but I'm sure that isn't an issue for phpdig. Thanks, Dale. Last edited by guinessec; 11-30-2004 at 02:29 PM.