PDA

View Full Version : how to change user-agent ?


123av
11-16-2003, 06:38 PM
is it possible to create user-agent modification ?
i want that my phpdig present as something like 'my.domain.tld search engine' and phpdig spider search on robots.txt not for phpdig but for 'my.domain.tld'

Rolandks
11-17-2003, 11:30 PM
User-agent is hard-coded in: admin/robot_functions.php
Line 452
@ini_set('user_agent','PhpDig/'.PHPDIG_VERSION.' (PHP; MySql)'."\r\n".phpDigMakeCookies($cookiesToSend,$path));

and
Line 549 ff:

function phpdigReadRobotsTxt($site)
....


-Roland-

jerrywin5
03-25-2004, 12:18 AM
I found the user agent defined in multiple places.

In admin/spider.php on line (approx.) 157

In libs/phpdig_functions on lines (approx.) 315, 384, 477, and 620

It would be nice if the user agent could be set from the config file as well as the URI for the robot information page.

Charter
03-25-2004, 12:35 AM
Hi. The GNU/GPL (http://www.gnu.org/copyleft/gpl.html) license, Version 2, does not require you to keep the PhpDig User-Agent link in the user agent, just reference in the package itself must stay for redistribution, but personally it would be nice if the user agent and link remained. ;)

unknown
04-08-2004, 01:53 AM
hi, when i spider my site awstats tell me "Unknown robot (identified by 'robot')"
have someone an idea where can i chance this in the script (robot_functions and spider have the correct user_agent)

when phpdig find an site awstats show the referrer "links from other websites (no search engine)"

Charter
04-10-2004, 03:35 PM
Hi. This is not a PhpDig specific issue. Rather, AWStats does not recognize PhpDig, like it would Google, other than to say it is an "Unknown robot (identified by 'robot')" because 'robot' is part of the user-agent for PhpDig. Same goes for the "links from other websites (no search engine)" message. If you want AWStats to show something else, you would need to edit the AWStats code.

unknown
04-10-2004, 04:16 PM
hi, awstats known some more bots/crawler.
from my awstats:
Googlebot (Google)
MSNBot
Alexa (IA Archiver)
WISENutbot (Looksmart)
Fast-Webcrawler (AllTheWeb)
Inktomi Slurp
IBM_Planetwide
SurveyBot
Jeeves
Voila
LinkWalker
MSIECrawler

it is maybe that all stats-scripts can read the ident from phpdig?

Charter
04-10-2004, 06:47 PM
Hi. PhpDig is not currently part of your AWStats detection capabilities so you would need to edit your AWStats code.


# AWSTATS ROBOTS DATABASE
#-------------------------------------------------------
# If you want to add robots to extend AWStats database detection capabilities,
# you must add an entry in RobotsSearchIDOrder_listx and RobotsHashIDLib.
#-------------------------------------------------------

# AWSTATS SEARCH ENGINES DATABASE
#-------------------------------------------------------
# If you want to add a Search Engine to extend AWStats database detection capabilities,
# you must add an entry in SearchEnginesSearchIDOrder and in SearchEnginesHashIDLib.
# An entry if known in SearchEnginesKnownUrl is also welcome.
#-------------------------------------------------------

unknown
04-11-2004, 01:44 AM
ah ;)
now i understand
thx

tryangle
04-16-2004, 08:31 AM
Hi,


# AWSTATS ROBOTS DATABASE
#-------------------------------------------------------
# If you want to add robots to extend AWStats database detection capabilities,
# you must add an entry in RobotsSearchIDOrder_listx and RobotsHashIDLib.
#-------------------------------------------------------

# AWSTATS SEARCH ENGINES DATABASE
#-------------------------------------------------------
# If you want to add a Search Engine to extend AWStats database detection capabilities,
# you must add an entry in SearchEnginesSearchIDOrder and in SearchEnginesHashIDLib.
# An entry if known in SearchEnginesKnownUrl is also welcome.
#-------------------------------------------------------


Is this how you go about "naming" the spider so it doesn't show up as "Unknown robot (identified by 'robot')" in the log files of the spidered sites?

Charter
04-20-2004, 09:40 AM
Hi. Yes, check the top of the files in the AWStats library directory for more info. Also, there is an AWStats support form here (http://awstats.sourceforge.net/awstats_support.html) with links to AWStats FAQ and documentation.

tryangle
04-21-2004, 04:29 AM
Thanks Again