Upgrade Bugzilla-ETL Scripts

RESOLVED FIXED

Status

()

bugzilla.mozilla.org
Infrastructure
RESOLVED FIXED
4 years ago
4 years ago

People

(Reporter: ekyle, Assigned: fubar)

Tracking

Firefox Tracking Flags

(Not tracked)

Details

(URL)

The Current ETL has bugs.  Please upgrade the code to this new version (which is currently waiting for review)

1) turn off the 10min cron
2) perform upgrade
3) execute full etl (bzetl --settings=settings.json --reset)
4) wait a long time
4) turn on cron

Step two may be complicated because I do not know where the ./resources directory was placed.  I changed the installation procedure to NOT include Pip because Pip makes dealing with resource files complicated.   Instead, git clone will be used (https://github.com/klahnakoski/Bugzilla-ETL#installation).  Future upgrades (using git pull origin) will be much smoother (you will not even need to i/o the cron!)

The only important part is to not loose your unique settings.json file

Both public and private ETL need to be upgraded
(Reporter)

Updated

4 years ago
No longer blocks: 952101
Depends on: 952101
(Assignee)

Comment 1

4 years ago
So there's a known issue with RedHat servers, github and proxies. We can go one of two routes:

1) Generate an ssh key for the system account on etl1 and clone via ssh (see https://help.github.com/articles/generating-ssh-keys)

2) Ask opsec to grant us direct access to github (bypass the proxies). They'll likely do this, but requires waiting on them to do the work.

Also, once the repo is cloned, how does one go about installing or otherwise using it? Run "python setup.py" ?
Assignee: server-ops-webops → klibby
(Reporter)

Comment 2

4 years ago
After 
> git clone https://github.com/klahnakoski/Bugzilla-ETL.git

Jump into directory that Git made
> cd Bugzilla-ETL

and ensure the requirments are installed
> pip install -r requirements.txt

to run, ensure your settings.json file is accessible
> cd Bugzilla-ETL
> pypy bzETL\bz_etl.py --settings=path_to_settings.json


https://github.com/klahnakoski/Bugzilla-ETL#installation
(Reporter)

Updated

4 years ago
Blocks: 922877
Status: NEW → RESOLVED
Last Resolved: 4 years ago
Resolution: --- → FIXED
Component: WebOps: Bugzilla → Infrastructure
Product: Infrastructure & Operations → bugzilla.mozilla.org
You need to log in before you can comment on or make changes to this bug.