1038596 - Update verify can check partials against wrong base version

Status: RESOLVED INVALID

(Release Engineering :: Release Automation: Other, defect)

Description

[Pete Moore [:pmoore][:pete]]

Off the bat: my knowledge in this area is limited - so I'll write my understanding of the situation, and happy for people to step in and correct me or fill in the gaps! :)

From IRC on 15 July 2014 in #releng (timestamps in CEST):

10:24:33 nthomas: I've only looked at one update verify fail for 31.0, but it's failed at the first attempt to patch a file. It turns out the ship-it has 29.0.1build1 in the list of partials, and we shipped 29.0.1build2
10:25:08 nthomas: so we built the partial using the wrong build, and it doesn't apply to 29.0.1 build2 when we run update verify
10:26:50 nthomas: same screw up with 28.0

The problem appears to be this:

In ship it, you can specify a version to generate partials against (e.g. 29.0.1build1) but if you shipped a later build, e.g. 29.0.1build2, by the time you get to Update Verify, it will try to patch 29.0.1build2 which it got from the "patcher configs" rather than 29.0.1build1 which is the version you specified in ship it.

It is not clear to me why when you specify a specific version (29.0.1build1) that later on the version is taken from the patcher configs, rather than respecting the version already specified. Joining the dots, I am guessing that 29.0.1build2 is a replacement for 29.0.1build1 and therefore only one build of 29.0.1 should be published at a time (in other words, as soon as build2 is created, build1 is somewhat "redundant"). In any case, I think if you specified a specific build number ("build1") that should be respected, even if it has been superseded. Or the details of the "from" version should be burned into the partial, so that it can only be applied to that specific "from" frozen version - and when we test the partial, we could then grab the partial, see which "from" version it should be applied against, then download that, and apply it, to validate it. In short, I see no reason that these frozen versions should get "altered on the the way" between specifying in ship it, and update verify running.

Comment 1

[[e-mail] (:bhearsum)]

The reason for this is complicated, but I'll try to sum it up briefly.

Each lineage of releases (eg, release, beta, esr) has a patcher config (https://github.com/mozilla/build-tools/blob/master/release/patcher-configs/mozRelease-branch-patcher2.cfg). As part of the "updates" step of every release, we append the current release's information to it. "append" is a key point here - we don't change any data from previous releases. You can see this in commits like https://github.com/mozilla/build-tools/commit/f22ad44b418c97d9ee6648b450e6d2044fba28e4.

The patcher config is used for AUS2 style update generation, and update verify config generation. This means that in cases where we have multiple build #s, it only contains information for the one that was shipped. For example, when we built 30.0build1 we got this commit, which added 30.0 to the patcher config: https://github.com/mozilla/build-tools/commit/5e3c3bf277802ee8f76fc1093cc380c42e722ad4 -- and then when we build 30.0 build2, it updated that block to reflect build2: https://github.com/mozilla/build-tools/commit/0f1c1b4cfa5e4a84dede7f877461758817fd6cc3

All the above is mainly a way of explaining that the patcher config should always contains information about the build # we shipped of each release. This is important, because it means the metadata in it points to things our users actually run. This means that when we generate and run our update verify configs based on the patcher config, they catch problems such as the one yesterday, where we generated partial updates for 29.0.1build1, when we should've generated them for 29.0.1build2.


Given the above, I think this bug is INVALID - the updates & update verify steps are doing exactly what they should do. There's no doubt it's confusing than Ship It asks you for build #s, and then steps like this ignore them. I think we should solve that by having Ship It not ask for build numbers anymore. bug 1038673 may be a part of this, but it will get a whole lot easier when bug 886522 is implemented.

Comment 2

[Pete Moore [:pmoore][:pete]]

Hi Ben,

Thanks for taking the time to explain this, and it makes perfect sense. I saw https://bugzilla.mozilla.org/show_bug.cgi?id=886522#c4 too, which covers it from the ship it side - perfect! :)

Pete