PhpDig.net

PhpDig.net (http://www.phpdig.net/forum/index.php)
-   How-to Forum (http://www.phpdig.net/forum/forumdisplay.php?f=33)
-   -   Indexing .info (http://www.phpdig.net/forum/showthread.php?t=1692)

christophe 01-02-2005 08:16 AM

Indexing .info
 
Hi people
I have a other problem today :confused:
My phpDig indexing not the domain with extension ".info"
Perhaps this domain don't exist in USA but in France they have many .info.
How do you indexing .info ?
Please.

vinyl-junkie 01-02-2005 10:33 AM

The domain extension shouldn't have anything to do with whether or not phpdig can index the site. It's more likely that the site blocks all but certain spiders, or that individual pages have robots exclusion in place.

What is the site you're trying to index?

christophe 01-02-2005 10:42 AM

it's that

http://www.aujardin.info/

vinyl-junkie 01-02-2005 10:58 AM

I was able to spider 3 pages on my test server before I stopped the indexing process. However, spidering this site was extremely slow.

What happened when you tried to spider it? Can you post your spider log?

christophe 01-02-2005 12:19 PM

I indexed 1 only page.
I don't understand sometimes PhpDig indexing 100 pages et sometimes 1 or 2.
It's strange but i play with

vinyl-junkie 01-02-2005 12:28 PM

What values did you set for "links per" and "search depth?" Remember what the admin page says:

Quote:

- Search depth of zero tries to crawl just that page regardless of links per
- Set links per depth to the max number of links to check at each depth
- Links per depth of zero means to check for all links at each seach depth

christophe 01-03-2005 09:13 AM

I can indexing the domain in .info but with the mod "addurl", the people doesn't can add a .info domain.

Charter 01-03-2005 03:12 PM

Sounds like a regex issue. Check with the the mod "addurl" author, edit the mod "addurl" code, or offer to pay someone to edit the mod "addurl" for you.

Dave A 01-08-2005 12:52 PM

Re indexing of Info domains.
 
The spider does index domains with that extension (.info)
I have used it to index quite a few myself so it may be the robots.txt file or htaccess that is blocking you.

christophe 01-09-2005 04:03 AM

Ah yes Perhaps (may be)
Thanks


All times are GMT -8. The time now is 05:37 PM.

Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2025, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.