Searching UBB Message Board

Beans · 08-03-2005, 02:21 PM

First let me say that I really dig PhpDig. FOr the people who wine about spending $5 for access to the grand poobahs sage advice, consider their time and effort to develop this product before you complain about $5!

Now on to my question....

I want to index a message board running UBB, and PHPDig just won't do it.

I thought at first it was because the /bboard/cgi-bin/ directory that contains the ultimate.cgi file was listed in the Robots.txt for this server, and the managment of this board removed the disallow for that directory.. but PHPDig still won't index the threads in the message board.

So now i'm wondering if it's because this particular message board requires you to be registered to see the threads? For whatever reason, PhpDig is able to index other message boards set up that way, but not this one.

So if it's the registration thing that is my obstacle.. is there a way to get PhpDig to act as me when indexing? (i.e. mimick a cookie so when it hits the website, that website thinks it's me, a registered user, browsing the fourms?)

Thanks in advance!

Charter · 08-03-2005, 05:14 PM

PhpDig sees what an unauthenticated user sees, though you can have PhpDig spider htaccess protected content by indexing http://username:password@www.domain.com although it's not recommended, as the username and password are sent in plain text. For some ideas on how to index session and cookie protected content, check out this thread.

Beans · 08-03-2005, 08:57 PM

Is there any way to fool the message board into thinking PhpDig is me using my browser.. i.e., imbed a cookie session ID in it?

If not, sounds like the only way i'll be able to index that message board is if they make it readable to un-registered users.

Charter · 08-04-2005, 08:18 AM

Cookies are domain specific treats. Both UBB and PhpDig would need to be on the same domain for cookie integration, and even then you'd probably be better off using a reserved variable like in this thread.

08-03-2005, 02:21 PM	#1
Beans Green Mole Join Date: Jul 2005 Posts: 3	Searching UBB Message Board First let me say that I really dig PhpDig. FOr the people who wine about spending $5 for access to the grand poobahs sage advice, consider their time and effort to develop this product before you complain about $5! Now on to my question.... I want to index a message board running UBB, and PHPDig just won't do it. I thought at first it was because the /bboard/cgi-bin/ directory that contains the ultimate.cgi file was listed in the Robots.txt for this server, and the managment of this board removed the disallow for that directory.. but PHPDig still won't index the threads in the message board. So now i'm wondering if it's because this particular message board requires you to be registered to see the threads? For whatever reason, PhpDig is able to index other message boards set up that way, but not this one. So if it's the registration thing that is my obstacle.. is there a way to get PhpDig to act as me when indexing? (i.e. mimick a cookie so when it hits the website, that website thinks it's me, a registered user, browsing the fourms?) Thanks in advance!

08-03-2005, 05:14 PM	#2
Charter Head Mole Join Date: May 2003 Posts: 2,539	PhpDig sees what an unauthenticated user sees, though you can have PhpDig spider htaccess protected content by indexing http://username:password@www.domain.com although it's not recommended, as the username and password are sent in plain text. For some ideas on how to index session and cookie protected content, check out this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

08-04-2005, 08:18 AM	#4
Charter Head Mole Join Date: May 2003 Posts: 2,539	Cookies are domain specific treats. Both UBB and PhpDig would need to be on the same domain for cookie integration, and even then you'd probably be better off using a reserved variable like in this thread. __________________ Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.

Similar Threads
Thread	Thread Starter	Forum	Replies	Last Post
Message for Rispbiz	jerrywin5	The Mole Hole	0	03-08-2005 09:30 AM
Error message	-GO-	How-to Forum	2	08-19-2004 11:13 PM
Welcome message in template	claudiomet	How-to Forum	3	08-17-2004 02:58 PM
Invision power board - spidering doesn't work?	george	Troubleshooting	14	12-05-2003 07:37 AM
This board not letting you do something?	Charter	Feedback & News	2	08-03-2003 11:54 PM

08-03-2005, 08:57 PM	#3
Beans Green Mole Join Date: Jul 2005 Posts: 3	Is there any way to fool the message board into thinking PhpDig is me using my browser.. i.e., imbed a cookie session ID in it? If not, sounds like the only way i'll be able to index that message board is if they make it readable to un-registered users.