1490094 - [try-stage] Configure tasks by release type not branch.

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[Tom Prince [:tomprince]]

Staging releases want to behave like a release on another branch. Switch the in-tree config to key off of release type, and set the that per-branch and allow configuring it on try.

Comment 1

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [taskgraph] Remove some dead code refering to `release_type`; r?bhearsum

Comment 2

[[e-mail] (:bhearsum)]

Comment on attachment 9007876 [details]
Bug 1490094: [taskgraph] Remove some dead code refering to `release_type`; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 3

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/93b5503a50b0
[taskgraph] Remove some dead code refering to `release_type`; r=bhearsum

Comment 4

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/93b5503a50b0

Comment 5

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Configure staging release resources via dedicated parameter; r?bhearsum

Currently, release resources such as balrog, bouncer and CDN urls are
configured by project. However, all production branches use one value and all
other branches use another. Rather than duplicate lists of projects, add a
helper to identify production release branches.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/3748dbd2539c
[release] Configure staging release resources via dedicated parameter; r=bhearsum

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 9008188 [details]
Bug 1490094: [release] Configure staging release resources via dedicated parameter; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 8

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/3748dbd2539c

Comment 9

[Jorg K (CEST = GMT+2)]

Attachment: 1490094-C-C-part.patch

Comment 10

[Jorg K (CEST = GMT+2)]

Comment on attachment 9008345 [details] [diff] [review]
1490094-C-C-part.patch

That's all I can do via pattern matching :-(

Comment 11

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/21c798a31bdf
Port bug 1490094: [release] Configure staging release resources via dedicated parameter. rs=bustage-fix

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/4658bcf16f8d
Port bug 1490094: Follow-up: Fix indentation issues. rs=white-space-only DONTBUILD

Comment 13

[Rob Lemley [:rjl]]

Comment on attachment 9008345 [details] [diff] [review]
1490094-C-C-part.patch

Review of attachment 9008345 [details] [diff] [review]:
-----------------------------------------------------------------

There's a number of places where by-project has been changed to by-release-level, yet we're still referring to comm-(beta|esr.*). I suspect those should be staging based on the M-C changes.
I don't think I flagged them all.

::: taskcluster/ci/release-bouncer-aliases/kind.yml
@@ +19,1 @@
>              comm-(beta|esr.*): scriptworker-prov-v1/tb-bouncer-v1

should this be "staging" now?

::: taskcluster/ci/release-bouncer-sub/kind.yml
@@ +15,1 @@
>              comm-(beta|esr.*): scriptworker-prov-v1/tb-bouncer-v1

staging?

::: taskcluster/ci/release-generate-checksums/kind.yml
@@ +65,1 @@
>                      comm-(beta|esr.*):

production?

@@ +65,4 @@
>                      comm-(beta|esr.*):
>                          stage_product: "thunderbird"
>                          bucket_name: "net-mozaws-prod-delivery-archive"
> +                    production:

staging?

::: taskcluster/ci/release-mark-as-shipped/kind.yml
@@ +21,2 @@
>              comm-(beta|esr.*): scriptworker-prov-v1/tb-shipit-v1
> +            production: scriptworker-prov-v1/tb-shipit-dev

production
staging

@@ +27,1 @@
>              comm-(beta|esr.*): ['project:comm:thunderbird:releng:ship-it:production']

production
staging

::: taskcluster/ci/release-update-verify-config/kind.yml
@@ +33,1 @@
>                  try-comm-central: "http://ftp.stage.mozaws.net/pub"

staging
should there be a default on these?

@@ +37,1 @@
>                  try-comm-central: "https://archive.mozilla.org/pub"

staging

@@ +41,1 @@
>                  try-comm-central: "https://aus5.stage.mozaws.net"

staging

Comment 14

[Jorg K (CEST = GMT+2)]

(In reply to Rob Lemley [:rjl] from comment #13)
> There's a number of places where by-project has been changed to
> by-release-level, yet we're still referring to comm-(beta|esr.*). I suspect
> those should be staging based on the M-C changes.
I was wondering about that. But the M-C changes had nothing of the form
  try:
or
  mozilla-(beta|esr.*)

Anyway, the damage is done and the patch has landed. It got Dailies going. So backing it out is not an option. I'll leave it for you to fix ;-)

Comment 15

[Jorg K (CEST = GMT+2)]

BTW, I'm happy to have this backed out completely if you want to start from scratch.

Comment 16

[Tom Prince [:tomprince]]

Comment on attachment 9008345 [details] [diff] [review]
1490094-C-C-part.patch

Review of attachment 9008345 [details] [diff] [review]:
-----------------------------------------------------------------

The only valid values for `release-level` are `production` and `staging`. `production` corresponds to any of comm-{central,beta,esr60} whereas `staging` corresponds to `try-comm-central`.

Comment 17

[Jorg K (CEST = GMT+2)]

Thanks. I hope our build/release engineer will now take care of this.

Comment 18

[Jorg K (CEST = GMT+2)]

Comment on attachment 9008345 [details] [diff] [review]
1490094-C-C-part.patch

Thanks for the clarification, Tom. I'll land a follow-up to fix the release levels. I've considered backing the previous patches out, but that would just be unnecessary churn to correct a few lines.

Comment 19

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/e4373d203bcc
Follow-up: fix incorrect release levels. r=me DONTBUILD

Comment 20

[Jorg K (CEST = GMT+2)]

Attachment: folded-patch-for-review.patch

Here's the sum of what I landed if you want to take another look.

Comment 21

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [taskgraph] Add a transform to skip tasks by release type; r?bhearsum

Comment 22

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [taskgraph] Change `release_type` to include more than just `rc`; r?bhearsum

Comment 23

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Configure staging release resources via dedicated parameter; r?bhearsum

Currently, release resources such as balrog, bouncer and CDN urls are
configured by project. However, all production branches use one value and all
other branches use another. Rather than duplicate lists of projects, add a
helper to identify production release branches.

Comment 24

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Remove per-project secondary update verify config variation; r?bhearsum

These tasks only run on rc release promotion steps, so there is no need to very
the configuration per-project.

Comment 25

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Use release_type to configure update-verify-config tasks; r?bhearsum

Comment 26

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Use release_type to configure update-verify tasks; r?bhearsum

Comment 27

[Tom Prince [:tomprince]]

Attachment: Bug 1490094: [release] Use release_type to configure balrog tasks; r?bhearsum

Comment 28

[Tom Prince [:tomprince]]

Comment on attachment 9008648 [details] [diff] [review]
folded-patch-for-review.patch

Review of attachment 9008648 [details] [diff] [review]:
-----------------------------------------------------------------

These changes all look correct.

Comment 29

[Jorg K (CEST = GMT+2)]

Comment on attachment 9008345 [details] [diff] [review]
1490094-C-C-part.patch

Problems of this patch amended in follow-up, landed in comment #19.

Comment 30

[[e-mail] (:bhearsum)]

Comment on attachment 9008823 [details]
Bug 1490094: [release] Remove per-project secondary update verify config variation; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 31

[[e-mail] (:bhearsum)]

Comment on attachment 9008822 [details]
Bug 1490094: [release] Configure staging release resources via dedicated parameter; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 32

[[e-mail] (:bhearsum)]

Comment on attachment 9008803 [details]
Bug 1490094: [taskgraph] Add a transform to skip tasks by release type; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 33

[[e-mail] (:bhearsum)]

Comment on attachment 9008820 [details]
Bug 1490094: [taskgraph] Change `release_type` to include more than just `rc`; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 34

[[e-mail] (:bhearsum)]

Comment on attachment 9008824 [details]
Bug 1490094: [release] Use release_type to configure update-verify-config tasks; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 35

[[e-mail] (:bhearsum)]

Comment on attachment 9008825 [details]
Bug 1490094: [release] Use release_type to configure update-verify tasks; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 36

[[e-mail] (:bhearsum)]

Comment on attachment 9008826 [details]
Bug 1490094: [release] Use release_type to configure balrog tasks; r?bhearsum

Ben Hearsum (:bhearsum) has approved the revision.

Comment 37

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/9f2efc6828d5
[taskgraph] Add a transform to skip tasks by release type; r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/f345b131a734
[taskgraph] Change `release_type` to include more than just `rc`; r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/9cb31823dcd9
[release] Remove per-project secondary update verify config variation; r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/708cf763c173
[release] Use release_type to configure update-verify-config tasks; r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/5070b8f0149a
[release] Use release_type to configure update-verify tasks; r=bhearsum
https://hg.mozilla.org/integration/autoland/rev/5749fce97426
[release] Use release_type to configure balrog tasks; r=bhearsum

Comment 38

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/9f2efc6828d5
https://hg.mozilla.org/mozilla-central/rev/f345b131a734
https://hg.mozilla.org/mozilla-central/rev/9cb31823dcd9
https://hg.mozilla.org/mozilla-central/rev/708cf763c173
https://hg.mozilla.org/mozilla-central/rev/5070b8f0149a
https://hg.mozilla.org/mozilla-central/rev/5749fce97426

Comment 39

[Jorg K (CEST = GMT+2)]

Attachment: C-C patch for porting changesets landed on 2018-09-18

This appears to be working, I did my best matching the patterns. Most likely this will need a follow-up. I added 'default' eight times where the FF files didn't have it, I'll point it out in the next comment.

Ideally of course we'd prepare a patch and have it reviewed and approved before the M-C patch even hits autoland/inbound.

Comment 40

[Jorg K (CEST = GMT+2)]

Comment on attachment 9009885 [details] [diff] [review]
C-C patch for porting changesets landed on 2018-09-18

Review of attachment 9009885 [details] [diff] [review]:
-----------------------------------------------------------------

::: taskcluster/ci/release-update-verify-config/kind.yml
@@ +49,5 @@
> +            by-release-type:
> +                beta: beta
> +                esr60: nonbeta
> +                nightly: beta
> +                default: null

(1) Added this.

@@ +62,1 @@
>                  default: null

(2) I believe this was removed in the FF changesets.

@@ +79,5 @@
> +                by-release-type:
> +                    beta: "beta-localtest"
> +                    esr60: "release-localtest"
> +                    nightly: "default"
> +                    default: null

(3-7): Added this five times for the five platforms.

::: taskcluster/ci/release-update-verify/kind.yml
@@ +43,1 @@
>                      default: "default"

(8) I believe this was removed in the FF changesets.

Comment 41

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/03d79b0cf048
Port changesets from bug 1490094 and bug 1487172. rs=bustage-fix

Comment 42

[Tom Prince [:tomprince]]

Comment on attachment 9009885 [details] [diff] [review]
C-C patch for porting changesets landed on 2018-09-18

Review of attachment 9009885 [details] [diff] [review]:
-----------------------------------------------------------------

This looks good. There are a couple of changes needed for the staging balrog instances (where the rules don't line up with production). I had these values in my try pushes, but hadn't pushed them anywhere else.

::: taskcluster/ci/release-balrog-scheduling/kind.yml
@@ +36,5 @@
> +                            default: []
> +                    staging:
> +                        by-release-type:
> +                            beta: [43]
> +                            esr60: [820]

This should be 729.

::: taskcluster/ci/release-balrog-submit-toplevel/kind.yml
@@ +47,5 @@
> +                            default: []
> +                    staging:
> +                        by-release-type:
> +                            beta: [43]
> +                            esr60: [820]

This should be 729.

Comment 43

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/743a00e211a5
Follow-up: Correct balrog rule numbers for Thunderbird's 'staging'. r=me

Comment 44

[Rob Lemley [:rjl]]

Attachment: Follow-up: Change release-type from esr60 to release

Thunderbird does not use "esr.*" as a release type.
See https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/decision.py#l95

Comment 45

[Tom Prince [:tomprince]]

Comment on attachment 9010974 [details] [diff] [review]
Follow-up: Change release-type from esr60 to release

Review of attachment 9010974 [details] [diff] [review]:
-----------------------------------------------------------------

This looks sensible.

::: taskcluster/ci/release-update-verify-config/kind.yml
@@ +59,5 @@
>                          default: "56.0b3"
> +                release:
> +                    by-platform:
> +                        win64.*: null
> +                        default: "52.0"

I can't see the context for this, but I'd guess that win64 probably wants to have 60.0 as a watershed.

Comment 46

[Rob Lemley [:rjl]]

Yeah, win64 can be set to 60.0. I'll make the necessary change.

Comment 47

[Rob Lemley [:rjl]]

Attachment: Follow-up: Change release-type from esr60 to release

Patch updated to set win64 watershed version to 60.0.

Thunderbird does not use "esr.*" as a release type.
See https://hg.mozilla.org/mozilla-central/file/tip/taskcluster/taskgraph/decision.py#l95

Comment 48

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/ec9dbe9537af
Follow-up: Change release-type from esr60 to release. f=tomprince, rs=jorgk

Comment 49

[Tom Prince [:tomprince]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/cfdb32105f09
https://hg.mozilla.org/releases/mozilla-esr60/rev/aa9208c0d808
https://hg.mozilla.org/releases/mozilla-esr60/rev/4ab9dc2373dd
https://hg.mozilla.org/releases/mozilla-esr60/rev/953d79799a7a
https://hg.mozilla.org/releases/mozilla-esr60/rev/e8927180f267
https://hg.mozilla.org/releases/mozilla-esr60/rev/17bf0eb3a035
https://hg.mozilla.org/releases/mozilla-esr60/rev/66b6da22dc7f
https://hg.mozilla.org/releases/mozilla-esr60/rev/76ed65607f04

Comment 50

[Jorg K (CEST = GMT+2)]

comm-esr60 uplift: I merged the six patches into two:
https://hg.mozilla.org/releases/comm-esr60/rev/02ceff96f1f87197133ab75db22239f566791139
https://hg.mozilla.org/releases/comm-esr60/rev/440a81faa54eb312e75d696af3cc6cbdac2ae5b9