1135702 - create {mozilla,comm}-esr38 branch

Status: RESOLVED FIXED

(Release Engineering :: Release Requests, defect)

Description

[[e-mail] (:bhearsum)]

Unlike previous ESR branches we shouldn't enable nightly builds or use "esrpre" style versions. See bugs 1135024 and 1133250 for background on those things.

Comment 1

[Rail Aliiev [:rail]]

ESR31 was created in bug 1012647 as a reference

Comment 2

[Rail Aliiev [:rail]]

It would be also great to revise the docs at https://wiki.mozilla.org/ReleaseEngineering/How_To/Create_new_ESR_branch

Comment 3

[Rail Aliiev [:rail]]

There a chance that we'll need to enable win64 on esr38

Comment 4

[Rail Aliiev [:rail]]

Attachment: esr38-tools.diff

Comment 5

[Rail Aliiev [:rail]]

Attachment: esr38-buildbot-configs-2.diff

checkconfig passes

Comment 6

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8598867 [details] [diff] [review]
esr38-tools.diff

Review of attachment 8598867 [details] [diff] [review]:
-----------------------------------------------------------------

Can we really drop comm-esr31 ? Is there going to be no overlap ?

Comment 7

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8598868 [details] [diff] [review]
esr38-buildbot-configs-2.diff

Review of attachment 8598868 [details] [diff] [review]:
-----------------------------------------------------------------

Similar question about dropping comm-esr31 without any overlap with esr38.

::: mozilla/release-firefox-mozilla-esr38.py.template
@@ +36,5 @@
>  releaseConfig['appVersion']          = '{{ appVersion }}'
>  releaseConfig['milestone']           = releaseConfig['appVersion']
>  releaseConfig['buildNumber']         = {{ buildNumber }}
>  releaseConfig['baseTag']             = '{{ baseTag }}'
> +releaseConfig['partialUpdates']      = {}

Please add a TODO so that we put the template back in for the first updates.

@@ +114,5 @@
>  releaseConfig['releaseChannel']        = 'esr'
>  releaseConfig['updateChannels'] = {
>      "esr": {
>          "versionRegex": r"^.*$",
>          "ruleId": 34,

The three ruleIDs will need to be adjusted to whatever we end up for the esr38 specific rules in Balrog. Later ?

::: mozilla/staging_release-firefox-mozilla-esr38.py
@@ +32,3 @@
>  releaseConfig['partialUpdates']      = {
>      '31.3.0esr': {
>          'appVersion': '31.3.0',

Might be worth emptying this out too.

Comment 8

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #6)
> Comment on attachment 8598867 [details] [diff] [review]
> esr38-tools.diff
> 
> Review of attachment 8598867 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Can we really drop comm-esr31 ? Is there going to be no overlap ?

This is what we did last time. See http://hg.mozilla.org/build/buildbot-configs/rev/9f0b1c086769

(In reply to Nick Thomas [:nthomas] from comment #7)
> Comment on attachment 8598868 [details] [diff] [review]
> esr38-buildbot-configs-2.diff
> 
> Review of attachment 8598868 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> Similar question about dropping comm-esr31 without any overlap with esr38.
> 
> ::: mozilla/release-firefox-mozilla-esr38.py.template
> @@ +36,5 @@
> >  releaseConfig['appVersion']          = '{{ appVersion }}'
> >  releaseConfig['milestone']           = releaseConfig['appVersion']
> >  releaseConfig['buildNumber']         = {{ buildNumber }}
> >  releaseConfig['baseTag']             = '{{ baseTag }}'
> > +releaseConfig['partialUpdates']      = {}
> 
> Please add a TODO so that we put the template back in for the first updates.

Sure.
 
> @@ +114,5 @@
> >  releaseConfig['releaseChannel']        = 'esr'
> >  releaseConfig['updateChannels'] = {
> >      "esr": {
> >          "versionRegex": r"^.*$",
> >          "ruleId": 34,
> 
> The three ruleIDs will need to be adjusted to whatever we end up for the
> esr38 specific rules in Balrog. Later ?

Good catch. I think this is what we need to do here:

1) create a new set of rules for esr with version > 38.0
2) whenever we ready to ship esr38 to other esr versions, drop the version restriction (or set to 31, if 24 is not ready) and delete old rules. 

Something like https://aus4-admin-dev.allizom.org/rules#esr%2038

Does it make sense?

> ::: mozilla/staging_release-firefox-mozilla-esr38.py
> @@ +32,3 @@
> >  releaseConfig['partialUpdates']      = {
> >      '31.3.0esr': {
> >          'appVersion': '31.3.0',
> 
> Might be worth emptying this out too.

I also found that the staging channel names are wrong.

Comment 9

[Rail Aliiev [:rail]]

Additionally I may need to drop win64 support.

Comment 10

[Kent James (:rkent)]

I don't know the full implications of dropping comm-esr31 in this bug, but there will certainly be significant overlap of comm-esr31 and comm-esr38. We'll be doing Thunderbird releases off of comm-esr31 as long as there is a viable mozilla-esr31.

Comment 11

[Rail Aliiev [:rail]]

(In reply to Kent James (:rkent) from comment #10)
> I don't know the full implications of dropping comm-esr31 in this bug, but
> there will certainly be significant overlap of comm-esr31 and comm-esr38.
> We'll be doing Thunderbird releases off of comm-esr31 as long as there is a
> viable mozilla-esr31.

Thunderbird release automation doesn't support multiple lines for the same channel (release). comm-esr38 is just a repo name, not a separate channel. In other words the current workflow doesn't support overlaps.

We did the same in bug 1015942 when switched from esr24 to esr31. It would be great if we continue following the same workflow for this version as well. Otherwise it would be hard to be ready for next week's releases - we have a bunch of them. :/

Comment 12

[Kent James (:rkent)]

As I understand it, in Thunderbird 31 there were initially significant regressions so that updates of existing Thunderbird 24 users was delayed until 31.2 or 31.3  I'm not quite sure how that looks from a channel perspective.

How does Firefox do this? Will you able to do a security update of a user to a new 31.x release after 38.0 is released?

I'm not advocating making a change to whatever has been done in the past though.

Comment 13

[Rail Aliiev [:rail]]

(In reply to Kent James (:rkent) from comment #12)
> As I understand it, in Thunderbird 31 there were initially significant
> regressions so that updates of existing Thunderbird 24 users was delayed
> until 31.2 or 31.3  I'm not quite sure how that looks from a channel
> perspective.

We didn't push the updates to the release channel in that case.

> How does Firefox do this? 

What we do in Firefox is a PITA and causes a lot of pain during initial setup and for all builds that overlap.
 
> Will you able to do a security update of a user to
> a new 31.x release after 38.0 is released?

Not if 38.0 is not pushed and there is a security release.

> I'm not advocating making a change to whatever has been done in the past
> though.

I would really prefer to keep the same workflow we had last time to avoid unpredicted issues. Releng is really busy these days with all these "urgent" projects, especially FTP migration. :(

Comment 14

[Rail Aliiev [:rail]]

Attachment: esr38-buildbot-configs-3.diff

* added new rule IDs that match version > 38.0
* dropped win64 support
* addressed the comments

Comment 15

[Rail Aliiev [:rail]]

Attachment: esr38-tools-1.diff

Dropped win64

Comment 16

[Rail Aliiev [:rail]]

Comment on attachment 8599488 [details] [diff] [review]
esr38-buildbot-configs-3.diff

Ignore these for now, I'm going to split FF and TB.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8599488 [details] [diff] [review]
esr38-buildbot-configs-3.diff

Review of attachment 8599488 [details] [diff] [review]:
-----------------------------------------------------------------

r+ with a few nits - release-thunderbird-comm-esr38.py.template and similar need the partials blanked out like Firefox has; and also updated balrog ruleIds.

Also, the version conditions in the Balrog rules will need to be '>=38.0' instead of '>38.0'.

Comment 18

[Rail Aliiev [:rail]]

Attachment: buildbot-configs-esr38-no-thunderbird.diff

I split the patches so we can switch to comm-esr38 whenever we are ready. They pass checkconfig on dev-master.

Comment 19

[Rail Aliiev [:rail]]

Attachment: buildbot-configs-esr38-thunderbird.diff

Comment 20

[Rail Aliiev [:rail]]

Attachment: tools-esr38-no-tb.diff

Comment 21

[Rail Aliiev [:rail]]

Attachment: tools-esr38-tb.diff

Comment 22

[Rail Aliiev [:rail]]

:rkent, the current plan is to leave TB release on esr31, regardless of FF channel. Whenever you are ready we can switch to comm-esr38. Would this help?

Comment 23

[Rail Aliiev [:rail]]

(In reply to Nick Thomas [:nthomas] from comment #17)
> Comment on attachment 8599488 [details] [diff] [review]
> esr38-buildbot-configs-3.diff
> 
> Review of attachment 8599488 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> r+ with a few nits - release-thunderbird-comm-esr38.py.template and similar
> need the partials blanked out like Firefox has; and also updated balrog
> ruleIds.

Thunderbird doesn't have overlaps like FF ESR, this is why those entries are valid.

> 
> Also, the version conditions in the Balrog rules will need to be '>=38.0'
> instead of '>38.0'.

Right... Fixing

Comment 24

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8599622 [details] [diff] [review]
tools-esr38-no-tb.diff

Review of attachment 8599622 [details] [diff] [review]:
-----------------------------------------------------------------

::: lib/python/mozilla_buildtools/test/test_release_info.py
@@ +32,5 @@
>          self.assertEquals('release-firefox-mozilla-release.py', got)
>  
>      def testThunderbird(self):
> +        got = getReleaseConfigName('thunderbird', 'comm-esr38')
> +        self.assertEquals('release-thunderbird-comm-esr38.py', got)

Technically this belongs in the other patch ;-)

Comment 25

[Rail Aliiev [:rail]]

Comment on attachment 8599619 [details] [diff] [review]
buildbot-configs-esr38-no-thunderbird.diff

https://hg.mozilla.org/build/buildbot-configs/rev/9eb93630dfc1

Comment 26

[Rail Aliiev [:rail]]

Comment on attachment 8599622 [details] [diff] [review]
tools-esr38-no-tb.diff

https://hg.mozilla.org/build/tools/rev/5bce3e748017

Comment 27

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9eb93630dfc1

Comment 28

[Rail Aliiev [:rail]]

staging treeherder got it: https://treeherder.allizom.org/#/jobs?repo=mozilla-esr38

Comment 29

[Rail Aliiev [:rail]]

Per IRC, we want to have some overlap for comm-esr31 and comm-esr38.

It didn't work from the first attempt - hitting builder limits for tst-linux64 :/

Comment 30

[Rail Aliiev [:rail]]

Attachment: comm-esr31+38-tools.diff

* add comm-esr38
* rename patcher and update verify configs from mozRelease to mozEsr31
* add empty esr38 patcher and update verify configs

Comment 31

[Rail Aliiev [:rail]]

Attachment: comm-esr31+38-buildbot-configs-2.diff

* Disable fig, which had no pushes since November to pass the builder limit errors
* add comm-esr38
* update patcher and update verify config names according to the tools patch
* update latest symlink names
* add new rules to handle future esr38 updates (until 31 EOL)
* disable updates/partials for esr38
* disable comm-esr31 nightlies
* release*.py files updated just to pass checkconfig/reconfig, they'll be regenerated later

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 8602131 [details] [diff] [review]
comm-esr31+38-buildbot-configs-2.diff

Review of attachment 8602131 [details] [diff] [review]:
-----------------------------------------------------------------

Fig is booked by [e-mail]. Can you please verify with her that she doesn't need it anymore before landing this?

::: mozilla/release-thunderbird-comm-esr31.py
@@ +127,5 @@
>  releaseConfig['updateChannels'] = {
>      "release": {
>          "versionRegex": r"^.*$",
>          "ruleId": 35,
> +        "patcherConfig": "mozEsr31-thunderbird-branch-patcher2.cfg",

I'm not sure this change makes sense. Whichever one of the branches will be shipping updates to the "release" channel should use mozRelease configs, shouldn't it? It makes a lot more sense to me to have update configs correspond to the channel they're working with rather than the branch they happen to built from...

Comment 33

[Rail Aliiev [:rail]]

I can rename the files back, but encoding the branch name into the file instead of the channel has some pluses. Both update lines submit to the same channel, even though the rule IDs are different. At some point, when we decide to ship 38 to all users, we'll just need to adjust the Balrog rules, without touching (renaming) the patcher/verify configs.

Comment 34

[[e-mail] (:bhearsum)]

(In reply to Rail Aliiev [:rail] from comment #33)
> I can rename the files back, but encoding the branch name into the file
> instead of the channel has some pluses. Both update lines submit to the same
> channel, even though the rule IDs are different. At some point, when we
> decide to ship 38 to all users, we'll just need to adjust the Balrog rules,
> without touching (renaming) the patcher/verify configs.

If you feel this strongly I'm not going to object, but I still think it's a bit confusing to use branch names instead of channels in patcher/update verify config filenames. When I see "mozEsr38" I think that the files are only relevant to esr38 releases - not _all_ esrs.

Comment 35

[Rail Aliiev [:rail]]

Comment on attachment 8602129 [details] [diff] [review]
comm-esr31+38-tools.diff

https://hg.mozilla.org/build/tools/rev/9a3ac70c978a

Comment 36

[Rail Aliiev [:rail]]

Comment on attachment 8602131 [details] [diff] [review]
comm-esr31+38-buildbot-configs-2.diff

https://hg.mozilla.org/build/buildbot-configs/rev/3c4fe0bde1d7

Comment 37

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/9eb93630dfc1

Comment 38

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/3c4fe0bde1d7

Comment 39

[Nick Thomas [:nthomas] (UTC+12)]

Attachment: [svn productdelivery] Fix up cron

Brings over the filename change for the patcher config. This is what manages the symlink at /pub/mozilla.org/thunderbird/nightly/latest-candidate-comm-release.

Comment 40

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8603037 [details] [diff] [review]
[svn productdelivery] Fix up cron

Committed revision 104217.

Comment 41

[Rail Aliiev [:rail]]

Attachment: tb38-buildbot-configs.diff

Enable updates for TB 38, patcher config patch incoming

Comment 42

[Rail Aliiev [:rail]]

Attachment: tb38-tools.diff

cp mozEsr31-thunderbird-branch-patcher2.cfg mozEsr38-thunderbird-branch-patcher2.cfg

Comment 43

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8604879 [details] [diff] [review]
tb38-tools.diff

It's possible there will be a problem bumping this, lets solve that later.

Comment 44

[Rail Aliiev [:rail]]

Comment on attachment 8604879 [details] [diff] [review]
tb38-tools.diff

https://hg.mozilla.org/build/tools/rev/51a262af62c3

Comment 45

[Rail Aliiev [:rail]]

Comment on attachment 8604877 [details] [diff] [review]
tb38-buildbot-configs.diff

https://hg.mozilla.org/build/buildbot-configs/rev/978825343150

Comment 46

[Rail Aliiev [:rail]]

Attachment: tools-add-38.0esr-patcher-config.diff

Comment 47

[Rail Aliiev [:rail]]

it's similar to http://hg.mozilla.org/build/tools/rev/f1ec46bab22f

Comment 48

[Nick Thomas [:nthomas] (UTC+12)]

Comment on attachment 8605608 [details] [diff] [review]
tools-add-38.0esr-patcher-config.diff

>diff --git a/release/patcher-configs/mozEsr38-branch-patcher2.cfg b/release/patcher-configs/mozEsr38-branch-patcher2.cfg
>+            details   https://www.mozilla.com/%locale%/firefox/38.0/releasenotes/
>+            </partials>

No opening <partials>, please add one.

>+        <release>
>+            <38.0esr>
>+                checksumsurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/24.0esr-candidates/build1/%platform%/%locale%/firefox-24.0esr.checksums
>+                completemarurl   http://stage.mozilla.org/pub/mozilla.org/firefox/nightly/24.0esr-candidates/build1/update/%platform%/%locale%/firefox-24.0esr.complete.mar

s/24\.0/38.0/g

>+                locales   ach af an ar as ast az be bg bn-BD bn-IN br bs ca cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gu-IN he hi-IN hr hsb hu hy-AM id is it ja linux win32 ja-JP-mac osx kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW

There's some platform goo in there, try this
ach af an ar as ast az be bg bn-BD bn-IN br bs ca cs cy da de dsb el en-GB en-US en-ZA eo es-AR es-CL es-ES es-MX et eu fa ff fi fr fy-NL ga-IE gd gl gu-IN he hi-IN hr hsb hu hy-AM id is it ja ja-JP-mac kk km kn ko lij lt lv mai mk ml mr ms nb-NO nl nn-NO or pa-IN pl pt-BR pt-PT rm ro ru si sk sl son sq sr sv-SE ta te th tr uk uz vi xh zh-CN zh-TW

r+ with those fixed up.

Comment 49

[Rail Aliiev [:rail]]

Comment on attachment 8605608 [details] [diff] [review]
tools-add-38.0esr-patcher-config.diff

remote:   https://hg.mozilla.org/build/tools/rev/651f86737204
remote:   https://hg.mozilla.org/build/tools/rev/066b81eef190

Tagged as well

Comment 50

[Rail Aliiev [:rail]]

Attachment: configs-enable-esr38-updates.diff

Comment 51

[Rail Aliiev [:rail]]

Comment on attachment 8605812 [details] [diff] [review]
configs-enable-esr38-updates.diff

https://hg.mozilla.org/build/buildbot-configs/rev/82d98710a67d

Comment 52

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/978825343150

Comment 53

[Rail Aliiev [:rail]]

In production: https://hg.mozilla.org/build/buildbot-configs/rev/82d98710a67d