finding dead links

manute · 01-10-2004, 05:34 PM

hi!

the problem that i got is not actually about phpdig i guess, but perhaps phpdig can help solve it.
i got quite a big site of about 1500 urls. unfortunately there are some dead links in it. first i manually deleted them after using the phpdig-spider, but after the next time of indexing, they were there again.
now that's getting on my nerves and i just can't find out where these dead links come from.
so now i thought phpdig at least has to "know" on which pages are some dead links. does anyone have an idea how to use phpdig to find out where they are?

Charter · 01-11-2004, 03:47 PM

Hi. I'm not sure if this is what you are looking to do, but you could echo the URL of the page. In robot_functions.php find the following code and try adding the line indicated below.

PHP Code:


			
if (ereg("HTTP/[0-9.]+ (([0-9])[0-9]{2})", $answer,$regs)) {

    if ($regs[2] == 2 || $regs[2] == 3) {

      $code = $regs[2];

    }

    elseif ($regs[1] >= 401 && $regs[1] <= 403) {

          $status = "UNAUTH";

          break;

    }

    else {

        $status = "NOFILE";

        echo "<br>No File: " . $url . "<br>"; // add this line

        break;

    }

}

manute · 01-14-2004, 03:00 PM

hey charter!

thanks for the code but that doesn't help me. i know the urls that are broken, because i see them in phpdig's admin menu.
anyway, thank you.

01-10-2004, 05:34 PM	#1
manute Orange Mole Join Date: Oct 2003 Location: hamburg, germany Posts: 52	finding dead links hi! the problem that i got is not actually about phpdig i guess, but perhaps phpdig can help solve it. i got quite a big site of about 1500 urls. unfortunately there are some dead links in it. first i manually deleted them after using the phpdig-spider, but after the next time of indexing, they were there again. now that's getting on my nerves and i just can't find out where these dead links come from. so now i thought phpdig at least has to "know" on which pages are some dead links. does anyone have an idea how to use phpdig to find out where they are?

01-11-2004, 03:47 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	Hi. I'm not sure if this is what you are looking to do, but you could echo the URL of the page. In robot_functions.php find the following code and try adding the line indicated below. PHP Code: `if (ereg("HTTP/[0-9.]+ (([0-9])[0-9]{2})", $answer,$regs)) { if ($regs[2] == 2 \|\| $regs[2] == 3) { $code = $regs[2]; } elseif ($regs[1] >= 401 && $regs[1] <= 403) { $status = "UNAUTH"; break; } else { $status = "NOFILE"; echo "<br>No File: " . $url . "<br>"; // add this line break; } }` __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Grep command for finding Saga zero length	Dave A	Troubleshooting	2	02-06-2006 02:00 PM
Using phpdig for finding copyright infringements	leto	How-to Forum	0	09-22-2005 12:31 AM
Spider not finding anything.	nvahalik	Troubleshooting	2	01-25-2005 01:41 PM
Database dead or dying	Dave A	Troubleshooting	1	12-15-2004 10:08 PM

01-14-2004, 03:00 PM	#3
manute Orange Mole Join Date: Oct 2003 Location: hamburg, germany Posts: 52	hey charter! thanks for the code but that doesn't help me. i know the urls that are broken, because i see them in phpdig's admin menu. anyway, thank you.