View Full Version : Working with Domains

02-26-2004, 01:26 AM

I am working with PhpDig and it is working fine. I am doing some tests in order to start to PhpDig some sites.
I was testing a site but would like to PhpDig only some subdirectories. The site is this:
I am using depth 20 to go as much as possible inside the site.
The problem is: the http://dollar.biz.uiowa.edu/~fil/Erico/
subdirectory has some pages linked to http://dollar.biz.uiowa.edu and PhpDig start to Dig in this link.
I think that if I am PhpDiging in http://dollar.biz.uiowa.edu/~fil/Erico/
this should be the domain and so anything found linked different from this domain 'http://dollar.biz.uiowa.edu/~fil/Erico/
' should not be included in the database.
Is that possible? Am I doing something wrong?
Any help will pe very appreciated.


Wagner Bazarin
:confused: :confused: :confused:

02-28-2004, 03:28 PM
Hi. You might try setting up a robots.txt file, or after crawling, exclude and/or delete certain directories from the admin panel.