PhpDig.net

Go Back   PhpDig.net > PhpDig Forums > Mod Submissions

Reply
 
Thread Tools
Old 07-17-2004, 11:49 AM   #1
b2l_grefix
Green Mole
 
b2l_grefix's Avatar
 
Join Date: May 2004
Posts: 9
custom depth of search per site in cgi (including urllist)

Hi,

since I use phpdig to spider several sites I wanted it to be possible to give a custom search depth per site in a urllist when spidering with a cron job.

So I made a few modifications to spider.php and robot_functions.php.

Thought this might be usefull for some off you, so I like to share it with you

IMPORTANT: modifications are made for version 1.8.3!

You need to add an extra column (site_limit) is to the table sites in the phpdig database sites

ALTER TABLE PREFIXsites ADD site_limit SMALLINT( 6 ) ;
------------------------------------------------------------------------------

example of list.txt
not specifying the depth or setting the depth to 0 will cause the spider to use the default depth as set in config.php

shell command:

Code:
#php -f [PHPDIG_DIR]/admin/spider.php list.txt
site1.com will be spidered with a depth of 2
site2.com and site3.com will use the default value as specified in the config.php file
site4.com will be spidered with a depth of 2

It can also be used to spider a single site

shell command:
Code:
#php -f [PHPDIG_DIR]/admin/spider.php http://host.mydomain.com depth
examples:
Code:
#php -f [PHPDIG_DIR]/admin/spider.php http://host.mydomain.com 5
this will spider with a depth of 5
Code:
#php -f [PHPDIG_DIR]/admin/spider.php http://host.mydomain.com
Code:
#php -f [PHPDIG_DIR]/admin/spider.php http://host.mydomain.com 0
If not specified or 0, then the default depth will be used as set up in config.php
Attached Files
File Type: zip phpdig_customdepth_cgi.zip (23.8 KB, 32 views)
__________________
Grefix
b2l_grefix is offline   Reply With Quote
Reply

Thread Tools

Posting Rules
You may not post new threads
You may not post replies
You may not post attachments
You may not edit your posts

BB code is On
Smilies are On
[IMG] code is Off
HTML code is Off
Forum Jump

Similar Threads
Thread Thread Starter Forum Replies Last Post
Help including search.php dhoyos Troubleshooting 2 02-02-2007 12:53 PM
Search Depth using cron hpg4815 How-to Forum 1 10-02-2006 09:16 AM
Changing the search depth Brain How-to Forum 1 03-17-2004 03:34 AM
Search Depth using PhpDig with command line bazarin How-to Forum 1 02-20-2004 06:53 AM
Add search depth limit to the sites table peter Mod Requests 0 01-03-2004 09:14 PM


All times are GMT -8. The time now is 01:49 PM.


Powered by vBulletin® Version 3.7.3
Copyright ©2000 - 2024, Jelsoft Enterprises Ltd.
Copyright © 2001 - 2005, ThinkDing LLC. All Rights Reserved.