PhpDig.net - View Single Post - Newbie on Domains: Yes or No Answer Please :)

new2dev · 03-01-2005, 11:03 PM

Hi!

Would it be possible to say, force Dig to crawl many domains, but only start in a sublevel of the domain, and never "crawl" out of that domain to offsite links?

For example

fetch URLS to pages in:

domain1.com/subdir
domain1.com/subdir/subdir...

domain2.com/subdir/subdir/...

but never return results like:

domain1.com
domain1.com/doc.html
domain1.com/unspecified-dir/
domain1.com/unspecified-dir/doc.html

I hope this makes sense.
Hoping to use dig to spyder specific content on many domains
but not crawl around too much

or jump onto "offsite" undefined
domains.

A yes or No would be appreciated!

03-01-2005, 11:03 PM	#1
new2dev Green Mole Join Date: Feb 2005 Posts: 4	Newbie on Domains: Yes or No Answer Please :) Hi! Would it be possible to say, force Dig to crawl many domains, but only start in a sublevel of the domain, and never "crawl" out of that domain to offsite links? For example fetch URLS to pages in: domain1.com/subdir domain1.com/subdir/subdir... domain2.com/subdir/subdir/... but never return results like: domain1.com domain1.com/doc.html domain1.com/unspecified-dir/ domain1.com/unspecified-dir/doc.html I hope this makes sense. Hoping to use dig to spyder specific content on many domains but not crawl around too much or jump onto "offsite" undefined domains. A yes or No would be appreciated!