675973 - design & implement any changes necessary for nightly build support in Balrog

Status: RESOLVED FIXED

(Release Engineering Graveyard :: Applications: Balrog (backend), defect, P3)

Description

[[e-mail] (:bhearsum)]

We chatted a lot about this last week and I'm not sure if we decided exactly what we're going to do. We need to support nightly builds, and allow for each build/repack to be pushed independently of others. As it stands right now, the format of the JSON blobs prevent us from being able to do that. IIRC we discussed two ways of fixing it:
1) Move the buildid from platform level to locale level. We could either allow for it at platform level still and have the locale level override, or require it at locale level
2) Add support for aliasing locales to another blob. Doing this would end up with very minimal JSON that only has a list of platforms/locales and the blobs they should point to.

I don't recall making a final decision, but I _think_ we were more in favour of moving the buildid.

Comment 1

[[e-mail] (:bhearsum)]

https://bugzilla.mozilla.org/show_bug.cgi?id=693593#c1 talks about some additional complexity around extv and appv. Basically, because updates to nightly builds come in on a per-locale basis, Balrog will give invalid versions for a period of time every time we bump version numbers (every 6 weeks). We probably need to move or allow overrides for extv and appv down to the locale level to work around this.

I also remembering somebody thinking that we should/need to do the same for hashFunction, but I don't remember the justification.....

Comment 2

[[e-mail] (:bhearsum)]

Going to look at this once the initial nightly build api lands in bug 693593.

Comment 3

[[e-mail] (:bhearsum)]

Attachment: make extv, appv, and buildID overridable at locale level

This turned out to be simpler than I thought it would be. Most of the interesting change is in identifyRequest and expandRelease, relating to resolving alias'. Now that we have the Blob classes, I thought it made sense to move some logic to them. Originally, I only had the getters for buildid, appv, and extv. But when I started adjusting expandRelease/identifyRequest to use them I realized it simplified things in those methods to implement a getter for the platform data, and to move the alias resolution there, too. I can see arguments both ways - what do you think? My original code is here: https://github.com/bhearsum/balrog/compare/master...e1cb6e1e0ea9a525423f8f2db49b77b97a38714e

Past the rule tests and unit tests, I did some additional testing locally by importing the dev database from 2 days ago, importing the latest snippets, and ensuring that my local instance served correct snippets.

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 582367 [details] [diff] [review]
make extv, appv, and buildID overridable at locale level

This looks great from a quick read. Will look more closely tomorrow. Please update doc/DATA_FORMAT too.

Comment 5

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 582367 [details] [diff] [review]
make extv, appv, and buildID overridable at locale level

>diff --git a/auslib/blob.py b/auslib/blob.py
>index b39c8b0..43f525e 100644
>--- a/auslib/blob.py
>+++ b/auslib/blob.py
>@@ -74,16 +74,19 @@ class ReleaseBlobV1(Blob):
>         'platforms': {
>             '*': {
>                 'alias': None,
>                 'buildID': None,

Maybe this is becoming the documentation, but lets update doc/DATA_FORMAT until it does. Otherwise great stuff!

Comment 6

[[e-mail] (:bhearsum)]

(In reply to Nick Thomas [:nthomas] (away until Jan 3rd) from comment #5)
> Comment on attachment 582367 [details] [diff] [review]
> move extv, appv, and buildID overridable at locale level
> 
> >diff --git a/auslib/blob.py b/auslib/blob.py
> >index b39c8b0..43f525e 100644
> >--- a/auslib/blob.py
> >+++ b/auslib/blob.py
> >@@ -74,16 +74,19 @@ class ReleaseBlobV1(Blob):
> >         'platforms': {
> >             '*': {
> >                 'alias': None,
> >                 'buildID': None,
> 
> Maybe this is becoming the documentation, but lets update doc/DATA_FORMAT
> until it does.

I was thinking about this too, but didn't want to abandon DATA_FORMAT for code that has no docstrings in it :). Maybe we can move the comments to blob.py at some point, or generate a doc based on the code - or something. In any case, definitely a different bug.

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 582367 [details] [diff] [review]
make extv, appv, and buildID overridable at locale level

Landed in: https://github.com/mozilla/balrog/commit/38539c01ec9012523a02bff194856e8f39cf61f7
Jenkins run: https://jenkins.mozilla.org/job/Balrog/45/

Comment 8

[[e-mail] (:bhearsum)]

mass component change