483302 - Update affiliates tools for SFx stage

Status: RESOLVED FIXED

(mozilla.org Graveyard :: Server Operations, task)

Description

[Alex Buchanan [:abuchanan]]

Attachment: affiliates archive install.sql

Hey,

I've made some updates to the way the Affiliates program counts/tracks it's points and I need IT's help setting up a couple things.  At this time, these affect stage only.

1) Create a new database for stage, called something like sfx_stage_affiliates_archive.  install.sql attached.

2) Run updates.sql on the SFx stage DB (the Drupal DB, not sfx_stage_affiliates_archive).  updates.sql to be attached

3) Check out the Affiliates log parse tools for stage.  This hasn't been done before, only on production.  These are located at...

http://svn.mozilla.org/projects/spreadfirefox.com/logs-parse/affiliates-download-counting/

 you'll need to set up config.pl and logmaker-config.php
- config.pl: should point to sfx_stage_affiliates_archive DB
- config.pl: mozcom_logdir => './mozcomlogs/'
- config.pl: dmo_logdir    => './dmologs/'

- logmaker-config: $sfx_db should point to the SFx stage DB

4) I've created a log generator for testing, I'd like it to run on cron, daily, for stage only.

`php -f logmaker.php`

The script will try to create ./mozcomlogs/ and ./dmologs/ and write log files to these directories, so make sure the script has sufficient permissions to do this.  Also, the log files will be small (and gzipped) but if you'd like to be clean, you could also set up something to delete any files in these folders *older than 2 days*

4) run.pl should run daily, after the logmaker.php

I'm on IRC in #spreadfirefox and #bmo if you have questions

-------------------
(sorry for the short notice, but I need this today in order to continue working this weekend)

Comment 1

[Alex Buchanan [:abuchanan]]

Attachment: SQL to update affiliates tables

Comment 2

[Alex Buchanan [:abuchanan]]

as an addendum to task #1, you'll need to add the archive DB URI to Drupal's settings.php

I've updated settings.php.dist, so you can svn up that and merge in the changes.  It's only a couple lines, located at lines 92-94 , the difference should be clear.

Comment 3

[Dave Miller [:justdave]]

Comment on attachment 367297 [details]
affiliates archive install.sql

hmm, silly Firefox doesn't know that text/* should be displayed in the browser? => text/plain

Comment 4

[Dave Miller [:justdave]]

First off, I want to apologize to you... was intending to get this done last night, and I dozed off before I got to it.  Today was completely buried in family activities so I couldn't get to it today until just now.

This is all done now.  The cron jobs are in place, however, the run.pl cron job is going to die horribly when it runs... it has dependencies on a perl module (Proc::Queue) that I can't easily get (it's not available as an RPM from our usual sources, and the staging machines don't have general net access, so I can't download it to there from CPAN).

If you can find a way to make it work without that module it'd be great, otherwise we'll need to work out something for it (build our own RPM or something) because we'll have the same problem trying to put it into production.

Comment 5

[Dave Miller [:justdave]]

Oh, I did set up the cron job so it would mail you with the output (if any).  You'll get the 'Can't locate Proc/Queue.pm in @INC' output in tonight's run.

Comment 6

[Dave Miller [:justdave]]

I put in a request with rpmforge (the place we usually get perl package RPMs from) for them to include Proc::Queue in the list of perl modules they build.  Usually they're good about adding stuff like that on request but I don't know how quickly it'll happen.

Comment 7

[Alex Buchanan [:abuchanan]]

Thanks! That's great news.

r23337, The script isn't actually using child processes currently, so I've removed the call to Proc::Queue

Although, that's been there and it's been running on production, so I'm not sure how that's been working.  Is it a staging only problem?  Anyway, it's gone now.

Comment 8

[Alex Buchanan [:abuchanan]]

Err, sorry, jumped the gun on marking this fixed.  I should wait until the changes have been pushed.  Are run.pl and related scripts auto updating?

Comment 9

[Dave Miller [:justdave]]

They're not autoupdating.  I can probably make them so they do.

I just poked around on the production server, and I can't find any evidence of this stuff being run there.

Now that the Proc::Queue thing is cleared up, it's now getting this error:

DateTime::TimeZone version 0.59 required--this is only version 0.46 at /usr/lib/perl5/vendor_perl/5.8.8/i386-linux-thread-multi/DateTime.pm line 47.

Comment 10

[Alex Buchanan [:abuchanan]]

CC'ing chizu because I've worked with him on Affiliates stuff in production, he'll know where it's located.

chizu, did you have to work around the problems in comments #4 and #9 to set this up?

DateTime is being used in the script, is it possible to update this package?  I'm not sure yet why version 0.59 is being required.

Comment 11

[chizu]

Production is at dm-stats01:/usr/local/bin/sfx-affiliates

dm-stats01 has DateTime 0.67 installed, and Proc:Queue 1.18. These were likely installed/upgraded for AUS/DMO processing, and probably using CPAN.

Comment 12

[Alex Buchanan [:abuchanan]]

(In reply to comment #11)
> Production is at dm-stats01:/usr/local/bin/sfx-affiliates
> 
> dm-stats01 has DateTime 0.67 installed, and Proc:Queue 1.18. These were likely
> installed/upgraded for AUS/DMO processing, and probably using CPAN.

chizu, thanks!

justdave, can we get DateTime 0.59+ on the needed box?  

As plan B, do either of you know how to generate date strings in Perl without DateTime?

Comment 13

[Dave Miller [:justdave]]

Date::Format is the package I've always used.

Comment 14

[Alex Buchanan [:abuchanan]]

r23364 removes DateTime in favor of Date::Format and Date::Parse

Comment 15

[Alex Buchanan [:abuchanan]]

Hey, just got the word that we need to have this done by tomorrow, so just making sure it's in the queue.

Let me know if you need anything, I think at the time all we need is an svn up on the script and confirmation that it's working and has no unmet dependencies.

Thanks!

Comment 16

[Dave Miller [:justdave]]

[root@mrapp-stage02 affiliates-download-counting]# svn up
U    run.pl
Updated to revision 23456.
[root@mrapp-stage02 affiliates-download-counting]# su - apache -s /bin/bash
-bash-3.2$ cd /data/build/affiliates-download-counting/
-bash-3.2$ perl run.pl
Files parsed:
mozcom-/data/build/affiliates-download-counting/mozcomlogs/access_2009-03-16-1.gz
dmo-/data/build/affiliates-download-counting/dmologs/access_20090316.1.gz

Looks good to me.