PDA

View Full Version : indexing defined domain suffix


vispa
02-22-2005, 02:50 AM
Quite a simple question really.

Is there anyway to limit the spidering to domain suffix that end .co.uk, .me.uk, gov.uk etc?

Charter
02-22-2005, 04:43 PM
http://www.phpdig.net/forum/showthread.php?t=1852

vispa
02-23-2005, 04:06 AM
Hi Charter,

I have tried that, but it still appears to be allowing .com/.net web sites etc to be spidered and indexed :/

This is what I have got (I am using 1.8.7).

define('FORBIDDEN_EXTENSIONS','\.(net|com|rm|ico|cab|swf|css|gz|z|tar|zip|t gz|msi|arj|zoo|rar|r[0-9]+|exe|bin|pkg|rpm|deb|bz2)$');

Charter
02-23-2005, 04:34 AM
Oops, forget what I said, as FORBIDDEN_EXTENSIONS works on path/file, not domain name. Did you mod PhpDig to allow it to index across different domains?