1138930 - Implement Funsize balrog submitter script

Status: RESOLVED FIXED

(Release Engineering :: General, defect)

Description

[Rail Aliiev [:rail]]

Attachment: balrog-submitter-partials-only-tools.diff

Funsize will need to submit partial data to balrog. The current implementation requires complete update information.

The patch relaxes this requirement and adds a script for partial submission. There are 2 highlights in the script:

1. Balrog credentials are set in the script environment. They will be set by Taskcluster via encrypted env variables (see http://docs.taskcluster.net/docker-worker/ Encrypted Environment Variables). I think this is better than baking the credentials into a docker image, but we should be careful dumping debug information to the logs.

2. A manifest used in the script looks like this:

{
  "ACCEPTED_MAR_CHANNEL_IDS": "firefox-mozilla-release", 
  "appName": "Firefox", 
  "branch": "mozilla-release", 
  "from_buildid": "20150217132925", 
  "from_mar": "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0b10-candidates/build1/update/linux-i686/ach/firefox-36.0b10.complete.mar", 
  "hash": "089b713674c84dbf0f12529b94748be57e8a61bbe0914a7f765b7ca9e8e49377aa908a3380dca126c2383126130886c59dde5ec0a05438f61c5030ed0bcaebd8", 
  "locale": "ach", 
  "mar": "Firefox-mozilla-release-36.0-20150217132925-20150219174507.partial.mar", 
  "platform": "linux", 
  "repo": "https://hg.mozilla.org/releases/mozilla-release", 
  "revision": "59eefd8af5d7", 
  "signed": true, 
  "size": 3905978, 
  "to_buildid": "20150219174507", 
  "to_mar": "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0-candidates/build1/update/linux-i686/ach/firefox-36.0.complete.mar", 
  "version": "36.0"
}

Comment 1

[[e-mail] (:bhearsum)]

Comment on attachment 8571940 [details] [diff] [review]
balrog-submitter-partials-only-tools.diff

Review of attachment 8571940 [details] [diff] [review]:
-----------------------------------------------------------------

The cli.py changes look pretty straightforward and sane, I have a couple of higher level questions/comments though.

(In reply to Rail Aliiev [:rail] from comment #0)
> Created attachment 8571940 [details] [diff] [review]
> balrog-submitter-partials-only-tools.diff
> 
> Funsize will need to submit partial data to balrog. The current
> implementation requires complete update information.
> 
> The patch relaxes this requirement and adds a script for partial submission.
> There are 2 highlights in the script:
> 
> 1. Balrog credentials are set in the script environment. They will be set by
> Taskcluster via encrypted env variables (see
> http://docs.taskcluster.net/docker-worker/ Encrypted Environment Variables).
> I think this is better than baking the credentials into a docker image, but
> we should be careful dumping debug information to the logs.

It kindof sucks to have this script use a different way of passing credentials than all of the rest. What do you think about using a credentials file like the other scripts? Is that possible with TC?

> 2. A manifest used in the script looks like this:
> 
> {
>   "ACCEPTED_MAR_CHANNEL_IDS": "firefox-mozilla-release", 
>   "appName": "Firefox", 
>   "branch": "mozilla-release", 
>   "from_buildid": "20150217132925", 
>   "from_mar":
> "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0b10-
> candidates/build1/update/linux-i686/ach/firefox-36.0b10.complete.mar", 
>   "hash":
> "089b713674c84dbf0f12529b94748be57e8a61bbe0914a7f765b7ca9e8e49377aa908a3380dc
> a126c2383126130886c59dde5ec0a05438f61c5030ed0bcaebd8", 
>   "locale": "ach", 
>   "mar":
> "Firefox-mozilla-release-36.0-20150217132925-20150219174507.partial.mar", 
>   "platform": "linux", 
>   "repo": "https://hg.mozilla.org/releases/mozilla-release", 
>   "revision": "59eefd8af5d7", 
>   "signed": true, 
>   "size": 3905978, 
>   "to_buildid": "20150219174507", 
>   "to_mar":
> "http://ftp.mozilla.org/pub/mozilla.org/firefox/candidates/36.0-candidates/
> build1/update/linux-i686/ach/firefox-36.0.complete.mar", 
>   "version": "36.0"
> }

One of the things that I really dislike about the current scripts is that they depend on external config files (release configs, buildbot properties). I find that it makes them hard to understand, and even harder to test. How do you feel about actually passing this information to the script rather than putting it in a manifest?

::: scripts/updates/funsize-balrog-submitter.py
@@ +52,5 @@
> +    submitter.run(
> +        platform=manifest["platform"], buildID=manifest["to_buildid"],
> +        productName=manifest["appName"], branch=manifest["branch"],
> +        appVersion=manifest["version"], locale=manifest["locale"],
> +        hashFunction='sha512', extVersion=manifest["version"],

Let's not hardcode "sha512" here. Make it an arg with a default of sha512, please.

Comment 2

[Rail Aliiev [:rail]]

Attachment: balrog-submitter-partials-only-tools.diff

We talked about this and decided that the library changes should go to this patch and the actual script will be part of funsize.

Comment 3

[Rail Aliiev [:rail]]

Comment on attachment 8572741 [details] [diff] [review]
balrog-submitter-partials-only-tools.diff

https://hg.mozilla.org/build/tools/rev/db91d2ba0bc5