Update affiliates tools for SFx stage

RESOLVED FIXED

Status

mozilla.org Graveyard
Server Operations
--
blocker
RESOLVED FIXED
9 years ago
3 years ago

People

(Reporter: abuchanan, Assigned: justdave)

Tracking

Details

(URL)

Attachments

(2 attachments)

(Reporter)

Description

9 years ago
Created attachment 367297 [details]
affiliates archive install.sql

Hey,

I've made some updates to the way the Affiliates program counts/tracks it's points and I need IT's help setting up a couple things.  At this time, these affect stage only.

1) Create a new database for stage, called something like sfx_stage_affiliates_archive.  install.sql attached.

2) Run updates.sql on the SFx stage DB (the Drupal DB, not sfx_stage_affiliates_archive).  updates.sql to be attached

3) Check out the Affiliates log parse tools for stage.  This hasn't been done before, only on production.  These are located at...

http://svn.mozilla.org/projects/spreadfirefox.com/logs-parse/affiliates-download-counting/

 you'll need to set up config.pl and logmaker-config.php
- config.pl: should point to sfx_stage_affiliates_archive DB
- config.pl: mozcom_logdir => './mozcomlogs/'
- config.pl: dmo_logdir    => './dmologs/'

- logmaker-config: $sfx_db should point to the SFx stage DB

4) I've created a log generator for testing, I'd like it to run on cron, daily, for stage only.

`php -f logmaker.php`

The script will try to create ./mozcomlogs/ and ./dmologs/ and write log files to these directories, so make sure the script has sufficient permissions to do this.  Also, the log files will be small (and gzipped) but if you'd like to be clean, you could also set up something to delete any files in these folders *older than 2 days*

4) run.pl should run daily, after the logmaker.php

I'm on IRC in #spreadfirefox and #bmo if you have questions

-------------------
(sorry for the short notice, but I need this today in order to continue working this weekend)
(Reporter)

Comment 1

9 years ago
Created attachment 367302 [details]
SQL to update affiliates tables
(Reporter)

Comment 2

9 years ago
as an addendum to task #1, you'll need to add the archive DB URI to Drupal's settings.php

I've updated settings.php.dist, so you can svn up that and merge in the changes.  It's only a couple lines, located at lines 92-94 , the difference should be clear.
Assignee: server-ops → justdave
Attachment #367297 - Attachment mime type: text/x-sql → text/plain
Comment on attachment 367297 [details]
affiliates archive install.sql

hmm, silly Firefox doesn't know that text/* should be displayed in the browser? => text/plain
First off, I want to apologize to you... was intending to get this done last night, and I dozed off before I got to it.  Today was completely buried in family activities so I couldn't get to it today until just now.

This is all done now.  The cron jobs are in place, however, the run.pl cron job is going to die horribly when it runs... it has dependencies on a perl module (Proc::Queue) that I can't easily get (it's not available as an RPM from our usual sources, and the staging machines don't have general net access, so I can't download it to there from CPAN).

If you can find a way to make it work without that module it'd be great, otherwise we'll need to work out something for it (build our own RPM or something) because we'll have the same problem trying to put it into production.
Oh, I did set up the cron job so it would mail you with the output (if any).  You'll get the 'Can't locate Proc/Queue.pm in @INC' output in tonight's run.
I put in a request with rpmforge (the place we usually get perl package RPMs from) for them to include Proc::Queue in the list of perl modules they build.  Usually they're good about adding stuff like that on request but I don't know how quickly it'll happen.
(Reporter)

Comment 7

9 years ago
Thanks! That's great news.

r23337, The script isn't actually using child processes currently, so I've removed the call to Proc::Queue

Although, that's been there and it's been running on production, so I'm not sure how that's been working.  Is it a staging only problem?  Anyway, it's gone now.
Status: NEW → RESOLVED
Last Resolved: 9 years ago
Resolution: --- → FIXED
(Reporter)

Comment 8

9 years ago
Err, sorry, jumped the gun on marking this fixed.  I should wait until the changes have been pushed.  Are run.pl and related scripts auto updating?
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
They're not autoupdating.  I can probably make them so they do.

I just poked around on the production server, and I can't find any evidence of this stuff being run there.

Now that the Proc::Queue thing is cleared up, it's now getting this error:

DateTime::TimeZone version 0.59 required--this is only version 0.46 at /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi/DateTime.pm line 47.
(Reporter)

Comment 10

9 years ago
CC'ing chizu because I've worked with him on Affiliates stuff in production, he'll know where it's located.

chizu, did you have to work around the problems in comments #4 and #9 to set this up?

DateTime is being used in the script, is it possible to update this package?  I'm not sure yet why version 0.59 is being required.

Comment 11

9 years ago
Production is at dm-stats01:/usr/local/bin/sfx-affiliates

dm-stats01 has DateTime 0.67 installed, and Proc:Queue 1.18. These were likely installed/upgraded for AUS/DMO processing, and probably using CPAN.
(Reporter)

Comment 12

9 years ago
(In reply to comment #11)
> Production is at dm-stats01:/usr/local/bin/sfx-affiliates
> 
> dm-stats01 has DateTime 0.67 installed, and Proc:Queue 1.18. These were likely
> installed/upgraded for AUS/DMO processing, and probably using CPAN.

chizu, thanks!

justdave, can we get DateTime 0.59+ on the needed box?  

As plan B, do either of you know how to generate date strings in Perl without DateTime?
Date::Format is the package I've always used.
(Reporter)

Comment 14

9 years ago
r23364 removes DateTime in favor of Date::Format and Date::Parse
(Reporter)

Comment 15

9 years ago
Hey, just got the word that we need to have this done by tomorrow, so just making sure it's in the queue.

Let me know if you need anything, I think at the time all we need is an svn up on the script and confirmation that it's working and has no unmet dependencies.

Thanks!
Severity: critical → blocker
[root@mrapp-stage02 affiliates-download-counting]# svn up
U    run.pl
Updated to revision 23456.
[root@mrapp-stage02 affiliates-download-counting]# su - apache -s /bin/bash
-bash-3.2$ cd /data/build/affiliates-download-counting/
-bash-3.2$ perl run.pl
Files parsed:
mozcom-/data/build/affiliates-download-counting/mozcomlogs/access_2009-03-16-1.gz
dmo-/data/build/affiliates-download-counting/dmologs/access_20090316.1.gz

Looks good to me.
Status: REOPENED → RESOLVED
Last Resolved: 9 years ago9 years ago
Resolution: --- → FIXED
Product: mozilla.org → mozilla.org Graveyard
You need to log in before you can comment on or make changes to this bug.