|
03-30-2004, 08:28 AM | #1 |
Orange Mole
Join Date: Mar 2004
Posts: 48
|
User Submitted URLs Mod Version 1
This mod allows users to submit URLs to be indexed. URLs are partially verified upon submission. The URLs are stored in a new table titled 'newsites' for approval. The admin can review all submissions and choose which to delete and which to index. URLs selected for indexing are placed in a text file for the spider to index either from a shell session or via a cron job.
Tested on Linux only but should work on a Windows server as well. If you have any problems, suggestions, and/or questions regarding this mod, please post them in this thread. All feedback welcome. |
03-30-2004, 08:33 AM | #2 |
Orange Mole
Join Date: Mar 2004
Posts: 48
|
File wasn't attached for some reason. Hopefully it will work this time.
Last edited by jerrywin5; 03-30-2004 at 08:42 AM. |
03-31-2004, 04:48 AM | #3 |
Green Mole
Join Date: Mar 2004
Posts: 8
|
Hmm, looks good, but i receive the following Error when i want to add an Url.:
Unknown column 'date_added' in 'field list' ?? Edit: And i get the same Error when i want to access the newurls.php! Last edited by firestarter; 03-31-2004 at 05:09 AM. |
03-31-2004, 06:50 AM | #4 |
Orange Mole
Join Date: Mar 2004
Posts: 48
|
User Submitted URLs Mod Version 1.1
Sorry about that. I added a field to the table and forgot to update DDL. Here is an updated version.
|
03-31-2004, 07:14 AM | #5 |
Green Mole
Join Date: Mar 2004
Posts: 8
|
No Prob
Ill check it and let you know - Thanks very much! Frank |
04-26-2004, 11:09 AM | #6 |
Green Mole
Join Date: Feb 2004
Posts: 6
|
Has anybody managed to get this working. Ive installed it but cant get anything to go into the newsites table.
|
04-26-2004, 02:22 PM | #7 | |
Green Mole
Join Date: Apr 2004
Location: North Wales
Posts: 11
|
Quote:
Dumb question.. did u submit a site using the addurl page? Would you be a bit more specific with the problem and perhaps we can solve it. snorkpants. |
|
04-26-2004, 11:41 PM | #8 |
Green Mole
Join Date: Feb 2004
Posts: 6
|
I did use the addurl page. But the new sites table in the database always stays empty. Any Ideas?
|
04-30-2004, 07:31 AM | #9 |
Orange Mole
Join Date: Mar 2004
Posts: 48
|
The mod uses the same connection file. Are you able to add records to the table manually? If so, it could be a matter of how php is set up on your server.
|
05-12-2004, 01:11 PM | #10 |
Green Mole
Join Date: May 2004
Posts: 23
|
This doesn't work if you have anything other than the default table prefix. My tables are phpdig prefixed so when adding a url I get the error that dbname.sites doesn't exist.. well of course not because it's phpdig_sites.
Anyway.. doesn't work because it doesn't use the prefix. |
05-13-2004, 12:18 AM | #11 |
Green Mole
Join Date: May 2004
Location: France
Posts: 8
|
Great, but needs some changes:
1- replace the short tag on first line by standard tag: replace <? with <?php 2- replace $HTTP_SERVER_VARS with $_SERVER 3- replace $HTTP_POST_VARS with $_POST 4- replace $HTTP_GET_VARS with $_GET $HTTP_*_VARS are not globals by default, that's also why some of you have problems with it. Since php 4.1.0 $_xxx should be used instead of $HTTP_*_VARS. Also, in the lasts php register_globals is set "off" by default for security reason. That's why $HTTP_*_VARS return empty values. |
08-19-2004, 11:06 AM | #12 |
Green Mole
Join Date: May 2004
Posts: 23
|
Does this work on 1.8.3?
|
08-23-2004, 01:04 PM | #13 |
Green Mole
Join Date: Aug 2004
Posts: 4
|
how to get the email working?
i will need to use my isp's smtp server and i cant figure it out in the php file
here are the problems it gives me: Warning: Failed to connect to mailserver, verify your "SMTP" setting in php.ini in c:\appserv\www\search\urlfiles\addurl.php on line 14 Warning: Cannot add header information - headers already sent by (output started at c:\appserv\www\search\urlfiles\addurl.php:14) in c:\appserv\www\search\urlfiles\addurl.php on line 15 |
08-27-2004, 12:34 PM | #14 |
Green Mole
Join Date: Jan 2004
Posts: 15
|
Problem with newurls.txt
Due to many problems trying to index urls from a text file I made this little script from JWSmythe's build.searchimages.pl.
This script will pull the url from the database and index it, delete it from the db, and then optimize tables. #!/usr/bin/perl use DBI; use MIME::Base64; $db = DBI->connect("DBI:mysql:database:localhost", username, 'password') || die "$!"; $source_query = "SELECT new_site_url FROM newsites "; $source = $db->prepare("$source_query") || die "$!, error on source prepare\n"; $source->execute || print "Error on source execute\n"; while (@curarray = $source->fetchrow_array){ $req_url = $curarray[0]; $req_url =~ s/\;//g; $outfile = $req_url; chop ($outfile); $outfile =~ s/\n//g; $outfile = "$outfile"; print "Indexing $req_url -> .....\n"; $sysstring = "php -f /path/to/admin/spider.php $req_url"; system(`$sysstring`); print "Finished Indexing $req_url -> ...Complete\n"; $db->do("DELETE FROM newsites WHERE new_site_url = '$curarray[0]'"); $db->do("OPTIMIZE TABLE newsites"); $db->do("OPTIMIZE TABLE tempspider"); }; Then I can run this script from shell or cron with this command with no problem. perl /path/to/cgi-bin/newurls.pl Note: If running cron be sure to allow enough time to index new urls before starting new cron. So don't set your cron up for evey minute.
__________________
Sometimes the shortest way home is the longest way around! Thank you PhpDig for a great search engine! www.2-surf.net |
08-29-2004, 02:56 PM | #15 |
Green Mole
Join Date: Feb 2004
Posts: 6
|
Would somebody please help me?
I am strugeling to get the script to spider the sites i have added automatically. Could someone provide me with an idiots guide ? Thanks |
|
|
Similar Threads | ||||
Thread | Thread Starter | Forum | Replies | Last Post |
um new user help | mspmsp | How-to Forum | 1 | 09-16-2004 06:35 PM |
User Titles | Charter | The Mole Hole | 0 | 01-30-2004 10:55 PM |
Mod Proxy (English version) | Iltud | Mod Submissions | 1 | 09-20-2003 11:56 AM |
Mod Proxy (Version française) | Iltud | Mod Submissions | 1 | 09-20-2003 11:53 AM |