PDA

View Full Version : How do i index my test server.


silverfish
03-29-2004, 11:12 AM
I want to index my localhost, which is just my ip right now, as it is a test server.

Ever time i try to index my IP, one level in.. It jsut indexes forever.. saying LOCKED over and over.. it never stops, and it isnt putting anything into the database.

If anyone has successfully indexed a folder in their localhost, please let me know how to do it!

Silver

Charter
03-29-2004, 12:37 PM
Hi. Perhaps something in this (http://www.phpdig.net/showthread.php?threadid=310) thread might help.

silverfish
03-29-2004, 12:39 PM
Hmm...

No... See im behind a firewall, but so is my server...So im just indexing locally...

It just says locked, but nothing is being written to the db, yet it loads and loads..

Charter
03-29-2004, 12:50 PM
Hi. Is anything getting into the tempspider table?

silverfish
03-29-2004, 12:51 PM
Ok..
i changed the 'USE_IS_EXECUTABLE_COMMAND' to 0 and it seemed to work in indexing..

But it wont spider.. it just indexed the first page, thats it..

Charter
03-29-2004, 01:09 PM
Hi. What search depth are you using?

silverfish
03-29-2004, 01:10 PM
It doesnt seem to matter...

I go from 1 all the way to 20 and it still only returns the first page.

:(

Charter
03-29-2004, 01:18 PM
Hi. Perhaps try the suggestions in this (http://www.phpdig.net/showthread.php?threadid=710) thread, and if there is a base href tag, then try the code in this (http://www.phpdig.net/showthread.php?postid=1633#post1633) post.

silverfish
03-29-2004, 01:37 PM
Still only indexes the first page.

:(

Charter
03-29-2004, 01:47 PM
Hi. Try uncommenting //print $answer."<br>\n"; in robot_functions.php and see what you get on index.

silverfish
03-29-2004, 01:50 PM
This is what i get when I spider:


Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://webdev/
Exclude paths :
- @NONE@
1:http://webdev/mambo/
(time : 00:00:08)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://webdev/mambo/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

silverfish
03-29-2004, 01:54 PM
If i can get this to work, I can post to the whole mambo community about this engine.. :)

Charter
03-29-2004, 02:01 PM
Hi. Doesn't Mambo already have PhpDig integrated? I thought it did, but maybe I'm just confused.

Anyway, what did you get when you uncommented //print $answer."<br>\n"; in the robot_functions.php file?

silverfish
03-29-2004, 02:04 PM
Opps...

here is what I got:

HTTP/1.1 404 Not Found
HTTP/1.1 200 OK
Date: Mon, 29 Mar 2004 23:03:34 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=3e3025a85c2bba6ea08f8512c863dc2c; expires=Tue, 30-Mar-2004 11:03:34 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Mon, 29 Mar 2004 23:03:34 GMT
Content-Type: text/html

HTTP/1.1 200 OK
Date: Mon, 29 Mar 2004 23:03:35 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=1ede113f1f6232445c38b9a20237c08f; expires=Tue, 30-Mar-2004 11:03:35 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Mon, 29 Mar 2004 23:03:35 GMT
Content-Type: text/html

HTTP/1.1 404 Not Found

--------------------------------------------------------------------------------
SITE : http://webdev/
Exclude paths :
- @NONE@
HTTP/1.1 200 OK
Date: Mon, 29 Mar 2004 23:03:40 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=1ffb0f6dac1a34f53e5f5b7da5899104; expires=Tue, 30-Mar-2004 11:03:40 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Mon, 29 Mar 2004 23:03:40 GMT
Content-Type: text/html

1:http://webdev/mambo/
(time : 00:00:08)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://webdev/mambo/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

Charter
03-29-2004, 02:23 PM
Hi. What happens when you add one of the following to your hosts file?

127.0.0.1 localhost
127.0.0.1 webdev/
127.0.0.1 webdev/mambo/

# or maybe this:

127.0.0.1 localhost
127.0.0.1 webdev.com/
127.0.0.1 webdev.com/mambo/

# or maybe this:

127.0.0.1 localhost
127.0.0.1 www.webdev.com/
127.0.0.1 www.webdev.com/mambo/

silverfish
03-29-2004, 02:36 PM
I am using apache with windows.

silverfish
03-29-2004, 02:49 PM
I cant index my other site outside this server.. but i cannot index my localhost site past the first page..

:(

Arghhh!!

Mike

silverfish
03-29-2004, 03:04 PM
Hmm i edited the hosts file, didnt seem to do much....

:(

silverfish
03-29-2004, 03:08 PM
Hmm i edited the hosts file, didnt seem to do much....

:(
Spidering in progress...
HTTP/1.1 404 Not Found
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 00:07:23 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=6f5949a767040f2205d08dd10ef22138; expires=Tue, 30-Mar-2004 12:07:23 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 00:07:23 GMT
Content-Type: text/html

HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 00:07:23 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=7884431717962635e6edaf2e0de86858; expires=Tue, 30-Mar-2004 12:07:24 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 00:07:24 GMT
Content-Type: text/html

HTTP/1.1 404 Not Found

--------------------------------------------------------------------------------
SITE : http://localhost/
Exclude paths :
- @NONE@
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 00:07:29 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=c6d9f45c7fee53937c186d09fe4aee4f; expires=Tue, 30-Mar-2004 12:07:29 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 00:07:29 GMT
Content-Type: text/html

1:http://localhost/mambo/
(time : 00:00:08)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://localhost/mambo/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

Charter
03-29-2004, 03:21 PM
Hi. Is there an index page in the mambo directory, or are you trying to index the directory content?

silverfish
03-29-2004, 03:23 PM
The content is created dynamically.. but the spider should spider the links.

Charter
03-29-2004, 08:15 PM
Hi. It looks like PhpDig isn't recognizing any links from the page, but I'm not sure why. :( What happens if you enter the site from a different location? What gets indexed then?

Charter
03-30-2004, 06:12 AM
Hi. Here's another thing to try.

In spider.php find:

if (isset($lien['newhost'])) {

and afterwards place:

echo "New Url: http://".$lien['newhost'].$lien['path'].$lien['file']."<br>";
echo "Old Url: ".$url."<br><br>";

When you index, what do you get?

silverfish
03-30-2004, 07:47 AM
This is what I get:

Spidering in progress...
HTTP/1.1 404 Not Found
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 16:42:54 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=2c5587f668574f2f226b5414477a485a; expires=Wed, 31-Mar-2004 04:42:54 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 16:42:54 GMT
Content-Type: text/html

HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 16:42:55 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=8fa61cdab071b4d8ae778fb946bab4aa; expires=Wed, 31-Mar-2004 04:42:55 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 16:42:55 GMT
Content-Type: text/html

HTTP/1.1 404 Not Found

--------------------------------------------------------------------------------
SITE : http://webdev/
Exclude paths :
- @NONE@
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 16:43:01 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=58dd11d18fa05c61810a8b2e3a909c69; expires=Wed, 31-Mar-2004 04:43:01 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 16:43:01 GMT
Content-Type: text/html

1:http://webdev/mambo/
(time : 00:00:08)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://webdev/mambo/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

silverfish
03-30-2004, 08:23 AM
Hmm it found 9 links.. not suire why only nine...


Spidering in progress...
HTTP/1.1 404 Not Found
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:32 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=a87906bd18548ebbec948831c7b706c8; expires=Wed, 31-Mar-2004 05:08:32 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 17:08:32 GMT
Content-Type: text/html

HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:33 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=cd87162a1bfaa95bf6605be2f3fa3710; expires=Wed, 31-Mar-2004 05:08:33 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 17:08:33 GMT
Content-Type: text/html

HTTP/1.1 404 Not Found

--------------------------------------------------------------------------------
SITE : http://webdev/
Exclude paths :
- @NONE@
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:38 GMT
Server: Apache/1.3.29 (Win32)
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Expires: Mon, 26 Jul 1997 05:00:00 GMT
Pragma: no-cache
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=1fc9a1d3dba9d95f30b4712b2de765fb; expires=Wed, 31-Mar-2004 05:08:38 GMT; path=/
Set-Cookie: mosvisitor=1
Last-Modified: Tue, 30 Mar 2004 17:08:38 GMT
Content-Type: text/html

1:http://webdev/mambo/
(time : 00:00:08)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=304011bcdee116fbd36707af98a36ed9; expires=Wed, 31-Mar-2004 05:08:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=5d335223ad6d46ec0c4d11d0ca6bc08c; expires=Wed, 31-Mar-2004 05:08:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=844ff72a35830e71c4a898576ce94115; expires=Wed, 31-Mar-2004 05:08:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=4b39d7aa40ed6b36f57183ac9a0c9fc5; expires=Wed, 31-Mar-2004 05:08:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=0a672831a7d09ca2ed4fac184fc86b3c; expires=Wed, 31-Mar-2004 05:08:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ New Url: http://www.200interchange.com/updates/update_02_04.html
Old Url: http://webdev/

New Url: http://www.200interchange.com/updates/update_02_04.html
Old Url: http://webdev/

New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


level 1...
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:50 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=b38bcc92d2841d104ff4a4776996cf8f; expires=Wed, 31-Mar-2004 05:08:50 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

2:http://webdev/mambo/?option=content&task=view&id=1&Itemid=2
(time : 00:00:20)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:52 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:52 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:52 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=2b1e654f67940b3c193c7495dddbb495; expires=Wed, 31-Mar-2004 05:08:52 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:52 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=dd57a856984a8e9cec3ec69a66571390; expires=Wed, 31-Mar-2004 05:08:52 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:52 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=269c8d06d9a366ed545d64fe6c76ba7f; expires=Wed, 31-Mar-2004 05:08:52 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

+ New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:57 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=7740cc0f4f0e4c3a25571037d7998fcd; expires=Wed, 31-Mar-2004 05:08:57 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

Duplicate of an existing document
3:http://webdev/mambo/?option=content&task=view&id=3&Itemid=2
(time : 00:00:26)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:59 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:08:59 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:04 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=fe19bc9dc3907b2069956fcc69a42974; expires=Wed, 31-Mar-2004 05:09:04 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

4:http://webdev/mambo/?option=content&task=view&id=4&Itemid=
(time : 00:00:33)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:05 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:05 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:10 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=47b90cb0be7fa8f6d576c9c36839f0af; expires=Wed, 31-Mar-2004 05:09:10 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

Duplicate of an existing document
5:http://webdev/mambo/?option=content&task=view&id=5&Itemid=
(time : 00:00:39)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:11 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:11 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:16 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=6bf30fcd99967c95c10e2d02d4460b67; expires=Wed, 31-Mar-2004 05:09:16 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

Duplicate of an existing document
6:http://webdev/mambo/?option=content&task=view&id=8&Itemid=
(time : 00:00:45)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:17 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:17 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


level 2...
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:27 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=edad5cb9e902a30b0b0f6cef927309b9; expires=Wed, 31-Mar-2004 05:09:27 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

Duplicate of an existing document
7:http://webdev/mambo/?option=content&task=view&id=7&Itemid=
(time : 00:00:56)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:28 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:28 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:33 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=f48dc83fa28d8c6ad50afe12fd5965d2; expires=Wed, 31-Mar-2004 05:09:33 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

8:http://webdev/mambo/?option=com_registration&task=lostPassword
(time : 00:01:02)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:35 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:35 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:40 GMT
Server: Apache/1.3.29 (Win32)
X-Powered-By: PHP/4.3.4
Set-Cookie: sessioncookie=bf329c036413e5e4f1c8d10f8515abc2; expires=Wed, 31-Mar-2004 05:09:40 GMT; path=/
Set-Cookie: mosvisitor=1
Content-Type: text/html

9:http://webdev/mambo/?option=com_registration&task=register
(time : 00:01:08)
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:41 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Thu, 04 Mar 2004 19:57:09 GMT
ETag: "0-1c12-40478a15"
Accept-Ranges: bytes
Content-Length: 7186
Content-Type: text/css
HTTP/1.1 200 OK
Date: Tue, 30 Mar 2004 17:09:41 GMT
Server: Apache/1.3.29 (Win32)
Last-Modified: Wed, 03 Mar 2004 22:41:40 GMT
ETag: "0-3ff5-40465f24"
Accept-Ranges: bytes
Content-Length: 16373
Content-Type: text/css
New Url: http://www.mamboserver.com/mambo/
Old Url: http://webdev/


No link in temporary table

--------------------------------------------------------------------------------

links found : 9
http://webdev/mambo/
http://webdev/mambo/?option=content&task=view&id=1&Itemid=2
http://webdev/mambo/?option=content&task=view&id=3&Itemid=2
http://webdev/mambo/?option=content&task=view&id=4&Itemid=
http://webdev/mambo/?option=content&task=view&id=5&Itemid=
http://webdev/mambo/?option=content&task=view&id=8&Itemid=
http://webdev/mambo/?option=content&task=view&id=7&Itemid=
http://webdev/mambo/?option=com_registration&task=lostPassword
http://webdev/mambo/?option=com_registration&task=register
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

Charter
03-30-2004, 08:44 AM
Hi. What did you do to get it to work?

silverfish
03-30-2004, 08:50 AM
I didnt do anything... it just worked..

Problem is, there are a lot more than just 9 links on that site.. any reason as to why it wouldnt index the whole thing?

Mike

Charter
03-30-2004, 09:00 AM
Hi. Did you happen to restart Apache, and then it worked?

PhpDig is written to crawl links within one domain. If you want to crawl across domains, then check out this (http://www.phpdig.net/showthread.php?threadid=658) thread.

silverfish
03-30-2004, 09:42 AM
Hmmm

It really isnt a domain, its a directory off my localhost...

Charter
03-30-2004, 09:52 AM
Hi. What changes did you make to your hosts file?

silverfish
03-30-2004, 10:21 AM
Man, I just dont understand...

It desnt dig links.. like in text.. my article titles.. etc... i hate this!

SITE : http://webdev/
Exclude paths :
- administrator/
- classes/
- components/
- editor/
- images/
- includes/
- language/
- mambots/
- media/
- modules/
- templates/
- uploadfiles/
1:http://webdev/
(time : 00:00:06)
No link in temporary table

--------------------------------------------------------------------------------

links found : 1
http://webdev/
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

silverfish
03-30-2004, 11:16 AM
ok...

IM perplexed.. I mean. I have no menu, just an article on my front page.. and it WILL NOT DIG THIS ARTICLE!!!

WHY???

ARghhh!!!

silverfish
03-30-2004, 11:21 AM
This is my current result.. WHy is it seeing them as duplicates?

Spidering in progress...

--------------------------------------------------------------------------------
SITE : http://webdev/
Exclude paths :
- administrator/
- classes/
- components/
- editor/
- images/
- includes/
- language/
- mambots/
- media/
- modules/
- templates/
- uploadfiles/
1:http://webdev/
(time : 00:00:06)
+
level 1...
2:http://webdev/?option=content&task=view&id=8&Itemid=
(time : 00:00:16)
+ + + + +
level 2...
Duplicate of an existing document
3:http://webdev/?option=content&task=view&id=7&Itemid=
(time : 00:00:27)

Duplicate of an existing document
4:http://webdev/?option=content&task=view&id=5&Itemid=
(time : 00:00:33)

Duplicate of an existing document
5:http://webdev/?option=content&task=view&id=1&Itemid=2
(time : 00:00:38)

Duplicate of an existing document
6:http://webdev/?option=content&task=view&id=3&Itemid=2
(time : 00:00:43)

Duplicate of an existing document
7:http://webdev/?option=content&task=view&id=4&Itemid=
(time : 00:00:49)

No link in temporary table

--------------------------------------------------------------------------------

links found : 7
http://webdev/
http://webdev/?option=content&task=view&id=8&Itemid=
http://webdev/?option=content&task=view&id=7&Itemid=
http://webdev/?option=content&task=view&id=5&Itemid=
http://webdev/?option=content&task=view&id=1&Itemid=2
http://webdev/?option=content&task=view&id=3&Itemid=2
http://webdev/?option=content&task=view&id=4&Itemid=
Optimizing tables...
Indexing complete !
--------------------------------------------------------------------------------
[Back] to admin interface.

Charter
03-30-2004, 11:58 AM
Hi. There is a thread here (http://www.phpdig.net/showthread.php?threadid=242) that talks about how PhpDig determines a duplicate page. As I cannot access your web content, I'm not sure that I can offer constructive help.

silverfish
03-30-2004, 12:38 PM
OK, when it says it found nine links...

BUT....

Why when i search for something in one of those links is it not there??? What does "found 9 links" mean if you cant go to them through the search?

Mike

Charter
03-30-2004, 12:41 PM
Hi. Look in your PhpDig text_content directory, view a file, and try a search on some words that appear in the file.

silverfish
03-30-2004, 12:48 PM
Nope cannot see any of the words from the page being referenced in the link on the content directory... :(

Charter
03-30-2004, 12:50 PM
Hi. Do you have the following set in the config file?

define('DISPLAY_SNIPPETS',true);

silverfish
03-30-2004, 01:12 PM
Yes, it is set to true.

Charter
03-30-2004, 01:14 PM
Hi. Are there any TXT files in the PhpDig text_content directory?

silverfish
03-30-2004, 03:19 PM
yup

Charter
03-30-2004, 07:43 PM
Hi. Do you see keywords in the keywords table? If so, try a search on one of the words in the keywords table. Do you have more than one install? Could it be that the search is looking in the wrong place?