Spider Issues

[LauzonRSD]

I am trying to spider my site, http://massmma.net

Latest stable version of phpDig is on the same domain.

I added my site to the index list, http://massmma.net

Now it says 1 host, but 0 pages.

DataBase status
Hosts : 1 Entries
Pages : 0 Entries
Index : 0 Entries
Keywords : 0 Entries
Temporary table : 0 Entries

I asked if I was on a load balanced server, and was told I was not.

Thanks

[LauzonRSD]

When I added my site to the index list, http://massmma.net, with a depth of 5, and Link Per of 0.

When I tell it to dig, it gives the display

Spidering in progress... [Stop spider]
Optimizing tables...
Indexing complete ! [Back] to admin interface.


Also, I can get it to Spider Another site, subfighter.com.

[Charter]

A test index of your site gave the following:

Spidering in progress... [Stop spider]
SITE : http://massmma.net/
Exclude paths :
- @NONE@
1:http://massmma.net/
(time : 00:00:08)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
level 1...
2:http://massmma.net/readMore.php?id=67
(time : 00:00:29)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
3:http://massmma.net/readMore.php?id=8
(time : 00:00:42)
+ + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
level 2...
links found : 3
http://massmma.net/
http://massmma.net/readMore.php?id=67
http://massmma.net/readMore.php?id=8
Optimizing tables...
Indexing complete ! [Back] to admin interface.

It looks like your site is hosted by WinWeb, and it's a guess, but I'd say WinWeb is a reseller, and if so, it could be that whatever servers they offer do have load balancing on them.

If I had to take a guess, I'd guess that WinWeb is reselling through HE (see similar DC pics here (http://www.he.net/tour_fremont.html)) and that HE does have load balancing on its servers (see dedicated server features here (http://www.he.net/dedicated_addfeatures.html)).

If you have access to your Hosts file, try adding something like the following, where the IP is that of your site:

127.0.0.1 massmma.net

[LauzonRSD]

Okay. I do not have access to my hosts file, because I have no SSH access. I have emailed them asking for it, but I'm sure they will tell me no. Is there anything else I can try to do?

[Charter]

Try indexing the following, where LIMIT_TO_DIRECTORY is set to true in the config file:

http://www.winwebhosting.com/~massmma/

Does it index for you? If so, after it is done, edit the site_url entry in the sites table and the path entries in the spider table to reflect the direct links to your site.

[LauzonRSD]

I just did that. When it adds the site, its showing as http://www.winwebhosting.com, and it doesnt index.

[Charter]

The domain name could be changed in the table, but given it still won't index, then short of getting your host to do something or buying another hosting account, preferrably *nix and without load balancing, the next best thing might be to install Apache, MySQL, PHP, and PhpDig on the machine in front of you (if you are on Windows, check out easyphp.org) and index from there. Afterwards, make a MySQL dump of the PhpDig tables and install the dump server-side, and then FTP over the local PhpDig files to the server, changing your database information in the connect.php file, so you can do searches server-side.

[LauzonRSD]

I got my host to give me shell access, but only for 24 hours. I made the file /etc/hosts (there was not one there). Is this what it should include?



127.0.0.1 localhost
127.0.0.1 massmma.net


Thanks... and in the mean time, I am trying what you said about indexing the site from another page. I would like to keep it all together and not have to go through indexing from somewhere else though.

[Charter]

The IP for massmma.net should be your site IP address:

127.0.0.1 localhost
xxx.xxx.xxx.xxx massmma.net

Check here (http://practice.chatserve.com/hosts.html) for some further instruction for various OS.

[LauzonRSD]

Okay.... and now is that their local private IP? (like 10.0.0.2) or their public IP (70.85.77.84)?

I believe it should be the private IP, but want to make sure. Its a major hassle... they gave me SSH access... but no write privileges, so Im fighting them on that now.

[Charter]

As the Hosts file is a mapping between names and IPs, it should probably be the IP such that http://massmma.net/ and http://xxx.xxx.xxx.xxx/ go to the same website.