1312433 - Improvements for balrog.submitter.api

Status: RESOLVED INCOMPLETE

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[Simon Fraser [:sfraser] ⌚️GMT]

There are several improvements to be done to the balrog python api

1. Python 3 compatibility
2. Basic tests
3. Move from hg.m.o/build/tools and bare python to a python module
4. More storage from hg.m.o to balrog repository on github

Scope: migration of api.py and not updates.py or cli.py at this stage.

Requires a stub to be left in place for cli.py to use.

Comment 1

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: puppet changes

Comment 2

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

Review commit: https://reviewboard.mozilla.org/r/88560/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/88560/

Comment 3

[[e-mail] (:bhearsum)]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review87682

Do you know if there's consumers of api.py that don't manage their own python packages? In the past, we've had things that would simply clone build/tools and run it. (This is why things like requests are in https://hg.mozilla.org/build/tools/file/tip/lib/python/vendor.) If we still have any of these, you'll need to place balrogclient into the vendor directory. r=me if there's none of those - I suspect the releaseduty folks will know if they still exist or not.

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review87682

Not sure I'm understanding this right. There's at least two situations I can think of that use the cli.py which at its turn use api.py:
1. funsize-balrog-submitter jobs in Release Promotion use https://hg.mozilla.org/mozilla-central/file/tip/release/docker/funsize-balrog-submitter that clones tools repo here https://hg.mozilla.org/mozilla-central/file/tip/release/docker/funsize-balrog-submitter/Dockerfile#l19 and use it here https://hg.mozilla.org/mozilla-central/file/tip/release/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py#l16

2. funsize-balrogworker (which is to be used in the nightly migration graph) that clones the tools repo here https://github.com/mozilla-releng/funsize-balrogworker/blob/master/Dockerfile#L24 and use it here https://github.com/mozilla-releng/funsize-balrogworker/blob/master/bin/balrogworker.py#L17

If you're thinking at the sole api.py consumers, I'd say we're safe on that side. I'd also double check with Rail too however.

Comment 5

[[e-mail] (:bhearsum)]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review87682

What about the Release Promotion jobs that run scripts such as https://hg.mozilla.org/build/tools/file/tip/scripts/updates/balrog-release-pusher.py and https://hg.mozilla.org/build/tools/file/tip/scripts/updates/balrog-release-shipper.py? Do they/will they be able to install balrogclient on their own?

Comment 6

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review87682

If there's no clear idea of where it gets run, perhaps it's easier to put it into the vendor directory and let the existing 'hg pull' sort it out, and then later on we can look at trying to record what's used where and how it's deployed, so we can make these decisions smoothly.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review87682

Fine with me. We can always remove it in the distant future.

Comment 8

[Simon Fraser [:sfraser] ⌚️GMT]

Attachment: Create copy of balrogclient module in vendor/ directory, for distribution. For bug 1312433,

Review commit: https://reviewboard.mozilla.org/r/88960/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/88960/

Comment 9

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/88560/diff/1-2/

Comment 10

[[e-mail] (:bhearsum)]

Comment on attachment 8805111 [details]
Create copy of balrogclient module in vendor/ directory, for distribution. For bug 1312433,

https://reviewboard.mozilla.org/r/88960/#review88112

All looks good to me as long you push these at the same time!

Comment 11

[[e-mail] (:bhearsum)]

Comment on attachment 8804644 [details]
Stub api.py for bug 1312433 - redirects module loads to new version installed as 'balrogclient'

https://reviewboard.mozilla.org/r/88560/#review88116

Comment 12

[[e-mail] (:bhearsum)]

Landed these as https://hg.mozilla.org/build/tools/rev/c41157cfe71773ca9763be088061b2e6343b075f

Comment 13

[Rail Aliiev [:rail]]

Attachment: Bug 1312433 - Regenerate docker image, bump funsize version

Review commit: https://reviewboard.mozilla.org/r/89804/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/89804/

Comment 14

[Simon Fraser [:sfraser] ⌚️GMT]

Comment on attachment 8806329 [details]
Bug 1312433 - Regenerate docker image, bump funsize version

https://reviewboard.mozilla.org/r/89804/#review89266

Comment 15

[Rail Aliiev [:rail]]

Comment on attachment 8806329 [details]
Bug 1312433 - Regenerate docker image, bump funsize version

remote:   https://hg.mozilla.org/build/puppet/rev/4ee7d8336125fd99498dd118912d53fd2dacc142
remote:   https://hg.mozilla.org/build/puppet/rev/5ddb82303f5988d1af873e7aa5f12ee80180ca90

Comment 16

[Rail Aliiev [:rail]]

backout:
remote:   https://hg.mozilla.org/build/puppet/rev/189ad2a70a2b29ba4701ad872c2d30a2f3b6fe73
remote:   https://hg.mozilla.org/build/puppet/rev/769f1f8d5a6bd98b8a3fac5dc18c14a0ea06b331

Comment 17

[[e-mail] (:bhearsum)]

Had to back this out for bustage: https://treeherder.mozilla.org/logviewer.html#?job_id=5470398&repo=mozilla-central#L77


We might need some sys.path manipulation somewhere?

Comment 18

[[e-mail] (:bhearsum)]

Once the swagger work (https://bugzilla.mozilla.org/show_bug.cgi?id=1336452) is done I think this might get replaced by an autogenerated (or mostly autogenerated) SDK. I'll leave this open for now, though.

Comment 19

[[e-mail] (:bhearsum)]

Simon, is there anything here you think is worth tracking still? If so, let's file as an Issue.

Comment 20

[Simon Fraser [:sfraser] ⌚️GMT]

It doesn't look like we ended up with an autogenerated library, but the code has been moved to https://github.com/mozilla-releng/scriptworker-scripts/tree/master/balrogscript/src/balrogscript/submitter and updated, so I think we're ok for the moment. It probably does need reworking at some point.