958039 - Upgrade Bugzilla-ETL Scripts (again)

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

The upgrade from Bug 952108 completed.  A "long run" analysis with real data has shown bugs in the ETL, again.  Please upgrade the application:

The instructions (https://github.com/klahnakoski/Bugzilla-ETL#upgrades) are:

1) cd ~/Bugzilla-ETL  # assuming this is the git clone directory
2) git pull origin master
3) pip install -r requirements.txt #not needed this time
4) Then delete the first_run_time file

The location of the first_run_time file can be found in settings.json file.  Since IT has a cron job that triggers every 10 min, the removal of the first_run_time file will make the next trigger perform a full ETL.

Both public and private ETL must be

Comment 1

[Kendall Libby [:fubar] (he/him)]

(In reply to Kyle Lahnakoski [:ekyle] from comment #0)

> Both public and private ETL must be

The suspense is killing me! They must be WHAT?!?! :)

I'll update this today.

Comment 2

[Kyle Lahnakoski [:ekyle]]

... updated

Comment 3

[Kendall Libby [:fubar] (he/him)]

Ok, updated and it's currently running full translations for both public and private (we'll see how well the VM copes with THAT!)

I've also shuffled things around a bit: the puppet config has been cleaned up, so it's easier to deal with and start/stop, plus I've also set it up so that we're deploying your code the same way we handle other webapp code. Basically, it's just:

you push new code to github, ask us for an update
we run a script on bugzillaadm which pulls from github and then pushes out to the etl node and we remove the first_run_time files

Once I get the docs on the wiki any of us should be able to handle it.

(still need to get pypy built and packaged; ran out of memory building it on our rpm build host last night. ugh.)