PhpDig.net - View Single Post

docphilz · 12-09-2003, 03:09 PM

Hi all,
first of all, be sure i've carefully read all the threads about this neverending problem.

So I've installed and set up phpdig 1.6.4 on my ISP site.
I actually don't know what php/apache/linux version is running :-/

Anyway, I can correctly crawl some sites (i.e. www.expertilog.com) very straight forward.

But when I try do crawl any site hosted at my isp, including my site, I get the famous :

SITE : http://www.netstudio.fr/
Exclude paths :
-
- @NONE@
No link in temporary table

--------------------------------------------------------------------------------

links found : 0
...Was recently indexed
Optimizing tables...
Indexing complete !

I did activate error trace in spider.php ( I'm a newbie to php).
I get several lines of :

Notice: Undefined variable: user_agent in /mnt/pro/client/939/742/searchengine/admin/robot_functions.php on line 564

But the fact is that i actually do not have any robots.txt file.
I tried both (with and without) and the result is the same.

The site i'd like to crawl is www.netstudio.fr .....

any comment or suggestion will be very appreciated !
:-)
cheers!

note : I'm aware about the \n\r issue, I had previously 1.6.2 with the same problem, and tried to fix it by editing the correct file, and then installed 1.6.4 which is supposed to fix it.

12-09-2003, 03:09 PM	#1
docphilz Green Mole Join Date: Dec 2003 Posts: 3	classical... Links found 0 Hi all, first of all, be sure i've carefully read all the threads about this neverending problem. So I've installed and set up phpdig 1.6.4 on my ISP site. I actually don't know what php/apache/linux version is running :-/ Anyway, I can correctly crawl some sites (i.e. www.expertilog.com) very straight forward. But when I try do crawl any site hosted at my isp, including my site, I get the famous : SITE : http://www.netstudio.fr/ Exclude paths : - - @NONE@ No link in temporary table -------------------------------------------------------------------------------- links found : 0 ...Was recently indexed Optimizing tables... Indexing complete ! I did activate error trace in spider.php ( I'm a newbie to php). I get several lines of : Notice: Undefined variable: user_agent in /mnt/pro/client/939/742/searchengine/admin/robot_functions.php on line 564 But the fact is that i actually do not have any robots.txt file. I tried both (with and without) and the result is the same. The site i'd like to crawl is www.netstudio.fr ..... any comment or suggestion will be very appreciated ! :-) cheers! note : I'm aware about the \n\r issue, I had previously 1.6.2 with the same problem, and tried to fix it by editing the correct file, and then installed 1.6.4 which is supposed to fix it.