PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > How-to Forum

Reply
 
Thread Tools
Old 01-13-2004, 06:15 PM   #1
tester
Green Mole
 
Join Date: Jan 2004
Posts: 10
Setting HTTP_USER_AGENT for spidering..

Hi Charter,

This is related to a problem that was posted in the troubleshooting section recently.

We have a website in which certain pages require authentication. This is performed using a function that is included on all protected pages. Our solution to allow the indexing of these pages was to check the HTTP_USER_AGENT from the request headers and allow the page to load using that. The problem is that when phpdig spiders the page, the user agent is always the default value of "PHP/4.2.2", no matter what we set it to be in the function phpdigTestUrl() in robotfunctions.php.

Is there a way to programmatically set User-agent to something secure so that the authentication mechanism is still dependable?

Thanks.
tester is offline   Reply With Quote
Old 01-14-2004, 10:11 AM   #2
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. What is user_agent set to in your PHP info? Are you crawling via shell or browser interface?
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Old 01-14-2004, 01:54 PM   #3
tester
Green Mole
 
Join Date: Jan 2004
Posts: 10
Smile

Hi Charter,

We are currently using the browser interface for crawling, with the intention of using shell later when we set up the indexing as a cron job.

In either case the spidering script is accessing the pages and providing a user agent of "PHP/4.2.2" (default before PHP version 4.3.0). The code in robot_functions.php allows the setting of the User-agent header, so why is this overridden?

The php.ini has nothing set for user_agent. Is there some other way to set the user_agent to our liking?
tester is offline   Reply With Quote
Old 01-15-2004, 06:37 AM   #4
Charter
Head Mole
 
Charter's Avatar
 
Join Date: May 2003
Posts: 2,539
Hi. What output do you get when you run the code snippet from php.net below?
PHP Code:
<?php
echo $_SERVER['HTTP_USER_AGENT'] . "<hr />\n";
$browser get_browser();
foreach (
$browser as $name => $value) {
   echo 
"<b>$name</b> $value <br />\n";
}
?>
__________________
Responses are offered on a voluntary if/as time is available basis, no guarantees. Double posting or bumping threads will not get your question answered any faster. No support via PM or email, responses not guaranteed. Thank you for your comprehension.
Charter is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Problem with setting the ABSOLUTE_SCRIPT_PATH wshobe Script Installation 2 07-18-2005 04:11 PM
Setting Up connect.php vinyl-junkie How-to Forum 1 08-15-2004 03:39 PM
Setting up phpdig in a subdomain vinyl-junkie Troubleshooting 1 05-29-2004 01:27 PM


All times are GMT -8. The time now is 09:23 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.