indexing defined domain suffix

vispa · 02-22-2005, 02:50 AM

Quite a simple question really.

Is there anyway to limit the spidering to domain suffix that end .co.uk, .me.uk, gov.uk etc?

Charter · 02-22-2005, 04:43 PM

http://www.phpdig.net/forum/showthread.php?t=1852

vispa · 02-23-2005, 04:06 AM

Hi Charter,

I have tried that, but it still appears to be allowing .com/.net web sites etc to be spidered and indexed :/

This is what I have got (I am using 1.8.7).

define('FORBIDDEN_EXTENSIONS','\.(net|com|rm|ico|cab|swf|css|gz|z|tar|zip|t gz|msi|arj|zoo|rar|r[0-9]+|exe|bin|pkg|rpm|deb|bz2)$');

Charter · 02-23-2005, 04:34 AM

Oops, forget what I said, as FORBIDDEN_EXTENSIONS works on path/file, not domain name. Did you mod PhpDig to allow it to index across different domains?

02-22-2005, 02:50 AM	#1
vispa Green Mole Join Date: Feb 2005 Posts: 4	indexing defined domain suffix Quite a simple question really. Is there anyway to limit the spidering to domain suffix that end .co.uk, .me.uk, gov.uk etc?

02-22-2005, 04:43 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	http://www.phpdig.net/forum/showthread.php?t=1852 __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

02-23-2005, 04:34 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Oops, forget what I said, as FORBIDDEN_EXTENSIONS works on path/file, not domain name. Did you mod PhpDig to allow it to index across different domains? __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Domain Aliases	exeter	How-to Forum	1	09-21-2005 07:49 AM
You are searching within domain.com	Slider	Mod Submissions	1	12-25-2004 09:48 PM
infinite loop when indexing sub-domain	janhsh	Bug Tracker	1	05-26-2004 04:55 PM
Indexing outside root domain	T3D	How-to Forum	5	03-14-2004 02:57 PM
one domain	xibalba	How-to Forum	1	03-12-2004 09:18 AM

02-23-2005, 04:06 AM	#3
vispa Green Mole Join Date: Feb 2005 Posts: 4	Hi Charter, I have tried that, but it still appears to be allowing .com/.net web sites etc to be spidered and indexed :/ This is what I have got (I am using 1.8.7). define('FORBIDDEN_EXTENSIONS','\.(net\|com\|rm\|ico\|cab\|swf\|css\|gz\|z\|tar\|zip\|t gz\|msi\|arj\|zoo\|rar\|r[0-9]+\|exe\|bin\|pkg\|rpm\|deb\|bz2)$');