1277871 - migrate existing funsize balrog worker to use new script worker functionality

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Kim Moir [:kmoir] ET]

https://pypi.python.org/pypi/scriptworker/

And

release/docker/funsize-balrog-submitter/scripts/funsize-balrog-submitter.py

Comment 1

[Kim Moir [:kmoir] ET]

Francis this is another bug you can look at

Comment 2

[Francis Kang]

What is the relation of this to bug 1244181?

Comment 3

[Kim Moir [:kmoir] ET]

for questions in comment 2

Comment 4

[Mihai Tabara [:mtabara]⌚️GMT]

Haven't touched bug 1244181 yet and, in the lights of the mozlondon conversations, some if its requirements and functionalities might have changed. I'll have a chat with Aki on Monday with respect to that bug, I'll be back with a more informed opinion by then. 

I'll leave the NI here to remind myself of this.

Comment 5

[Mihai Tabara [:mtabara]⌚️GMT]

See Aki's comment in bug 1244181.

Comment 6

[Francis Kang]

https://github.com/hfkang/funsize-balrog-scriptworker 

https://tools.taskcluster.net/task-inspector/#QL4qrbzUQEGvTBEhmvqHNQ/

I migrated the funsize-balrog submitter to run inside scriptworker, all of which is contained in a docker container. Right now it's pulling configuration and settings from the taskdef, environment variables, and cmd line args. I'd like to simplify it and centralize the config, but I'm not sure how this would be deployed, nor what the protocol is for handling secret credentials for S3 and balrog. 

Also it's currently encountering sporadic connection issues when POSTing data to my local balrog instance, so I'll have to do some testing with an external installation to see if the problem is just my machine.

Comment 8

[Francis Kang]

I've discussed deployment with bhearsum and mostlygeek, since balrog is moving over to cloudops. It seems the easiest option would be to run the scriptworker on the same machine, as it's not resource intensive.

Comment 9

[Francis Kang]

We moved the repo over to mozilla-releng today, and are currently enabling taskcluster hooks https://github.com/mozilla-releng/funsize-balrogworker. Deployment is planned to be on the CloudOps stack alongside balrog itself, which means it'll be using their Dockerflow for CI.

Comment 10

[Francis Kang]

https://github.com/mozilla-releng/funsize-balrogworker/commit/c5f951be7b0008637810b6a5809cd76859177365

https://tools.taskcluster.net/task-graph-inspector/#VjpDgCuAQG6ISsSTUkLk5Q/Beo9dHWUS1G9zu0bDRqrcw/0

Balrogworker is now building along a similar mechanism to balrog! It's currently uploading the docker images to my personal account, but that can be switched over once a new owner is found.

Comment 11

[Francis Kang]

Release style partials are live for only 1 week so I'm currently waiting for a Release promotion to be kicked off to test for that. I also plan to have this handle fennec builds once the release process for those has been finalized.

Comment 12

[Kim Moir [:kmoir] ET]

Francis: a release promotion build was kicked off last night but not sure if it's at the stage you need yet.

https://github.com/mozilla/releasewarrior/blob/master/releases/firefox-beta-48.0b9.md

Comment 13

[Francis Kang]

https://tools.taskcluster.net/task-graph-inspector/#T_7LS8enRLCjHjBY1ukTWw/ceR-vppOScqzalhvpdruVw/0 

Set up a CI test on taskcluster

Comment 14

[Chris Cooper [:coop] (he/him)]

Mihai: is there an update on deployment progress here?

Comment 15

[Mihai Tabara [:mtabara]⌚️GMT]

WIP in puppet to deploy balrogworker in our releng infra.  Bug 1289822 tracks that, will add more stuff as I hit tangiable milestones.

Comment 16

[Mihai Tabara [:mtabara]⌚️GMT]

I think this is done now as we have successful balrogworker publishing release blobs to our staging Balrog instance.
@Coop - any reason not to close this?