PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Troubleshooting

Reply
 
Thread Tools
Old 01-18-2005, 02:59 PM   #1
Dave A
Purple Mole
 
Dave A's Avatar
 
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
Indexing dynamically generated web pages

I wonder if there is a way of getting Phpdig to index dynaically generated web pages?
Each time I try to spider a web site that has dynanic pages generation it would appear that the spider doesn't find any content and can't index it. Perhaps there may be a few things within the configuration files that need amending?
So if any one has any ides could they please post an answer to the forum.

Many thanks
from Dave Downunder
Dave A is offline   Reply With Quote
Old 01-18-2005, 03:57 PM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
What version of PhpDig are you using? Would you provide an example link?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 01-18-2005, 06:10 PM   #3
Dave A
Purple Mole
 
Dave A's Avatar
 
Join Date: Aug 2004
Location: North Island New Zealand
Posts: 170
Relpy

Hi Charter,
firstly many thanks for getting back to me so quickly.
One example of the type of dynamic files I can't seem to index can be found via www.hastings.co.nz
The spider visits and tries for a few moments then replies with Indexed 0 files and then no link found in Temporary folder.
I have contacted the people who designed the web site and they have said that each page is dynamically generated.
Thanks for your help with this, typing is a little hard because I had a couple of cataract ops yesterday and things seem just a little fuzzy around the edges until the swelling has gone.

Many regards

Dave Andrews
Dave A is offline   Reply With Quote
Old 01-18-2005, 06:47 PM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
What version of PhpDig are you using?

If you use the latest version, you should see the following type output:

Spidering in progress... [Stop spider]
SITE : http://www.hastings.co.nz/
Exclude paths :
- @NONE@
1:http://www.hastings.co.nz/
(time : 00:00:08)

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml/
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
+
level 1...

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
2:http://www.hastings.co.nz/editable/welcome.shtml
(time : 00:00:20)

No link in temporary table
links found : 2
http://www.hastings.co.nz/
http://www.hastings.co.nz/editable/welcome.shtml
Optimizing tables...
Indexing complete ! [Back] to admin interface.

It has nothing to do with dynamic files. That site is giving 403s, meaning forbidden, not allowed, go away.

Hope your eyes feel better soon.
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 02-01-2005, 11:03 AM   #5
Paul D. Buck
Green Mole
 
Join Date: Jan 2005
Location: Sacramento
Posts: 8
Quote:
Originally Posted by Charter
What version of PhpDig are you using?

If you use the latest version, you should see the following type output:

Spidering in progress... [Stop spider]
SITE : http://www.hastings.co.nz/
Exclude paths :
- @NONE@
1:http://www.hastings.co.nz/
(time : 00:00:08)

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml/
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
+
level 1...

HTTP/1.1 403 Forbidden - http://www.hastings.co.nz/editable/welcome.shtml
See http://www.w3.org/Protocols/rfc2616/rfc2616-sec10.html for explanation.
2:http://www.hastings.co.nz/editable/welcome.shtml
(time : 00:00:20)

No link in temporary table
links found : 2
http://www.hastings.co.nz/
http://www.hastings.co.nz/editable/welcome.shtml
Optimizing tables...
Indexing complete ! [Back] to admin interface.

It has nothing to do with dynamic files. That site is giving 403s, meaning forbidden, not allowed, go away.

Hope your eyes feel better soon.
Ok, My question becomes, what links is it not getting? In other words, what has to change in the phpDig to find out which links are failing? I have had these failures on pages where all the links are internal to my site (as far as *I* can tell) but I am getting these 403 rejections too ...
Paul D. Buck is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Extracting search results and using them in your own web pages ciaran@clissman Mod Submissions 1 11-26-2005 10:14 AM
Indexing stops after a few pages Sibona Troubleshooting 1 05-03-2005 10:27 AM
is it real to inrease indexing time with web interface? zaartix How-to Forum 1 07-14-2004 08:13 PM
converted from html pages to php pages now no pages will index!!! help!! bigals Troubleshooting 24 04-01-2004 09:34 AM
Understanding logs from web indexing kenazo How-to Forum 1 03-15-2004 11:06 AM


All times are GMT -8. The time now is 12:10 AM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.