952108 - Upgrade Bugzilla-ETL Scripts

Status: RESOLVED FIXED

(bugzilla.mozilla.org :: Infrastructure, defect)

Description

[Kyle Lahnakoski [:ekyle]]

The Current ETL has bugs.  Please upgrade the code to this new version (which is currently waiting for review)

1) turn off the 10min cron
2) perform upgrade
3) execute full etl (bzetl --settings=settings.json --reset)
4) wait a long time
4) turn on cron

Step two may be complicated because I do not know where the ./resources directory was placed.  I changed the installation procedure to NOT include Pip because Pip makes dealing with resource files complicated.   Instead, git clone will be used (https://github.com/klahnakoski/Bugzilla-ETL#installation).  Future upgrades (using git pull origin) will be much smoother (you will not even need to i/o the cron!)

The only important part is to not loose your unique settings.json file

Both public and private ETL need to be upgraded

Comment 1

[Kendall Libby [:fubar] (he/him)]

So there's a known issue with RedHat servers, github and proxies. We can go one of two routes:

1) Generate an ssh key for the system account on etl1 and clone via ssh (see https://help.github.com/articles/generating-ssh-keys)

2) Ask opsec to grant us direct access to github (bypass the proxies). They'll likely do this, but requires waiting on them to do the work.

Also, once the repo is cloned, how does one go about installing or otherwise using it? Run "python setup.py" ?

Comment 2

[Kyle Lahnakoski [:ekyle]]

After 
> git clone https://github.com/klahnakoski/Bugzilla-ETL.git

Jump into directory that Git made
> cd Bugzilla-ETL

and ensure the requirments are installed
> pip install -r requirements.txt

to run, ensure your settings.json file is accessible
> cd Bugzilla-ETL
> pypy bzETL\bz_etl.py --settings=path_to_settings.json


https://github.com/klahnakoski/Bugzilla-ETL#installation