1491371 - Figure out how to pin thunderbird (comm-*) release branches that interacts nicely with actions as hooks.

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Tom Prince [:tomprince]]

Thunderbird is currently pinning revision of mozilla-* by adding `GECKO_HEAD_REV` to `.taskcluster.yml`. Since Bug 1415868 switched to using hooks for actions, the hooks encode a hash of .taskcluster.yml. That means that every time that a pin is updated, hooks need to be regenerated. This isn't scaleable.

Thus, we need to find a way to pin versions that doesn't involve changing `.taskcluster.yml`.

Comment 1

[Tom Prince [:tomprince]]

Talking with :dustin, we came up with a few thoughts on how this could be done. They all involved creating a new file in comm-central that has the pinned version, but differed in what looks at that file.

1) Teach mozilla-taskcluster[1] about the new file, and have it populate the context with contents of it when processing .taskcluster.yml
2) Teach `run-task`[2] to look for the file before checking out repositories (behind a flag of some kind). There is a bit of a chicken and egg issues, in that comm-central gets checked out after mozilla-central, so the would need to query hg.mozilla.org directly for the file. 
3) Add a helper script to the decision-task image that wraps `run-task`, queries hg.mozilla.org and populates `GECKO_HEAD_REV`
before calling run-task.


[1] https://github.com/taskcluster/mozilla-taskcluster/
[2] https://searchfox.org/mozilla-central/source/taskcluster/scripts/run-task

Comment 2

[Tom Prince [:tomprince]]

This should be resolved before Bug 1485680 gets uplifted to esr60, or merged to mozilla-beta.

Comment 3

[Dustin J. Mitchell [:dustin] (he/him)]

mozilla-taskcluster will soon(?) be replaced with a hook, and hooks are not sufficiently expressive to support (1).

Comment 4

[Pete Moore [:pmoore][:pete]]

Hey Tom, are you still interested in this?

Comment 5

[Tom Prince [:tomprince]]

Yes. This is going to become critical for Thunderbird, as the release promotion action is going to become a hook, and without this, the pinning that TB does will require releng intervention to create a new hook for many releases.

Comment 6

[Rob Lemley [:rjl]]

Discussed in IRC, will be implementing something like:

<tomprince> rjl: My thinking was for a new script that is in the decision task image, that bascially does data = json.loads(urllib.get(os.environ['COMM_HEAD_REPOSITORY']/.../os.environ['COMM_HEAD_REV']/raw-file/pin-data.json); env = os.environ.copy(); env['GECKO_HEAD_REV'] = data['GECKO_HEAD_REV']; os.exec(sys.argv[1:])

Comment 7

[Rob Lemley [:rjl]]

Attachment: Bug 1491371 - Pin comm branches to mozilla revs without .taskcluster.yml. r?dustin

run-task-wrapper runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of the comm repository,
".gecko_ref.yml"

The current pinning method of updating ".taskcluster.yml" is no longer
supported.

Comment 8

[Rob Lemley [:rjl]]

This is a WIP of the wrapper script for run-task. As suggested in comment 6, it will be added to the decision task image. The idea is that it will look at the COMM_HEAD_REPOSITORY and COMM_HEAD_REF environment variables that are set by Taskcluster and download a configuration file from that repository that contains the necessary information to set the GECKO_* environment variables that run-task needs.

In this revision, the downloaded file is named ".gecko_ref.yml" and will go in the root of the comm-* repositories. I opted to use YAML over JSON because I find it easier to work with.

defaults:
    gecko_base: mozilla-unified

comm-central:
    gecko_head: mozilla-central
    gecko_ref: default

comm-beta:
    gecko_head: mozilla-beta
    gecko_ref: abb8a4bb66a742bc1e31c3e2a2956bedf52be665

comm-esr60:
    gecko_head: mozilla-esr60
    gecko_ref: abb8a4bb66a742bc1e31c3e2a2956bedf52be665

try-comm-central:
    gecko_head: mozilla-central
    gecko_ref: default

It's written so a single file could be used for all of the comm repositories; That could be overkill. I've tried to keep it simple: just repository names and revisions.

The "command:" section of .taskcluster.yml gets a line added at the top:

        command:
          **- /builds/worker/bin/run-task-wrapper**
          - /builds/worker/bin/run-task
          - '--vcs-checkout=/builds/worker/checkouts/gecko'
          - '--sparse-profile=build/sparse-profiles/taskgraph'
          - '--comm-checkout=/builds/worker/checkouts/gecko/comm'
          - '--'
          - bash
          - -cx

The "GECKO_BASE_REPOSITORY", "GECKO_HEAD_REPOSITORY", and "GECKO_HEAD_REF" can get removed from .taskcluster.yml.
A new environment variable, "PROJECT_ENV_PREFIX" needs to be set. For Thunderbird repositories, it's just "COMM". This is used to construct the environment variable names "COMM_HEAD_REPOSITORY" and "COMM_HEAD_REF". The idea there being that there could be other "subprojects" that need this functionality.

I've only done some very basic testing locally.

Comment 9

[Magnus Melin [:mkmelin]]

(In reply to Rob Lemley [:rjl] from comment #8)

It's written so a single file could be used for all of the comm repositories; That could be overkill.

Wouldn't that make it hard to compare (i.e. diff) what needs to be done for the various versions?

Comment 10

[Rob Lemley [:rjl]]

Attachment: Bug 1491371 - Add config file for pinning to Mozilla repositories. r?darktrojan

Modifying .taskcluster.yml to pin comm-* repositories to m-* repo/rev's is
not supported going forward. This file is the new canonical location for
those pinnings.

Comment 11

[Rob Lemley [:rjl]]

Attachment: Bug 1491371 - Update decision task image to support comm-task-env. r?dustin

run-task-wrapper runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of a subproject's repository,
such as "comm-central".

Updates:

• add run-task-wrapper
• add python 3.5 (run-task dependency)
• add pyyaml

Comment 12

[Rob Lemley [:rjl]]

Attachment: Bug 1491371 - Update .taskcluster.yml to run comm-task-env r?dustin

Prepends/wraps "run-task" execution with "comm-run-task-wrapper". It will
download the configuration file containing the Gecko h.m.o. repository
information and set the appropriate environment variables so "run-task" can
run.

Comment 13

[Aki Sasaki (not active)]

Any timeframe for landing this? We're ready to land bug 1485680.

Comment 14

[Rob Lemley [:rjl]]

Per my conversation with Dustin earlier, this should be ready to go. I am personally not comfortable doing the decision task image update myself, so I've asked him to do that piece.

D16783 adds the wrapper script to m-c, then D16934 is the changes to Dockerfile/system-setup.sh to include that script.
I bumped the VERSION file at one point, as the docs said to do that early in the process.

Dustin, if you could build and deploy the new decision image incorporating those changes, I should be able to complete the rest as it's just the c-c changes. I'll be happy to coordinate timing with you. I am also unclear on what needs to be done with image hash updates that need to happen in various places.

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/58dec282b0bc
Pin comm branches to mozilla revs without .taskcluster.yml. r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/770808176c5c
Update decision task image to support comm-task-env. r=dustin

Comment 16

[Oana Pop-Rus]

https://hg.mozilla.org/mozilla-central/rev/58dec282b0bc
https://hg.mozilla.org/mozilla-central/rev/770808176c5c

Comment 17

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=2476ec6c8ed36f4cece6756d39a1e145bbc14e20

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=7b2ffc9d86e5c49129800736baa8b0114815c2e2

Comment 19

[Dustin J. Mitchell [:dustin] (he/him)]

That second try push failed with

[taskcluster 2019-03-21 17:14:11.427Z] === Task Starting ===
[setup 2019-03-21T17:14:11.779Z] run-task started in /
usage: run-task [-h] [--user USER] [--group GROUP]
                [--gecko-checkout GECKO_CHECKOUT]
                [--gecko-sparse-profile GECKO_SPARSE_PROFILE]
                [--comm-checkout COMM_CHECKOUT]
                [--comm-sparse-profile COMM_SPARSE_PROFILE]
                [--fetch-hgfingerprint]
run-task: error: unrecognized arguments: --vcs-checkout=/builds/worker/checkouts/gecko --sparse-profile=build/sparse-profiles/taskgraph
[taskcluster 2019-03-21 17:14:12.025Z] === Task Finished ===

It looks like the work in bug 1512188 was only partly finished, and in particular .taskcluster.yml was never updated. I'll re-open to finish that.

Comment 20

[Dustin J. Mitchell [:dustin] (he/him)]

Hm, I'll just solve it here actually.

Comment 21

[Dustin J. Mitchell [:dustin] (he/him)]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=13a52824b2be61aa2731ef2f92f6dde9ed108115

Comment 22

[Dustin J. Mitchell [:dustin] (he/him)]

Attachment: Bug 1491371, 1512188 - use new decision task image, with new options r=tomprince

Comment 23

[Dustin J. Mitchell [:dustin] (he/him)]

Rob, you should be good to go (assuming that sticks and hits mozilla-central)

Comment 24

[Tom Prince [:tomprince]]

Attachment: Bug 1491371: Update decision task image;

Comment 25

[Tom Prince [:tomprince]]

Attachment: Bug 1491371: [comm-task-env] Pass wrapped command name as argv[0]; r?dustin

Comment 26

[Tom Prince [:tomprince]]

jorgk, can you get the comm patches landed and uplifted to all branches. I don't think there are any dependencies on m-c based patches that need to be uplifted. The .gecko_rev.yml will need to be adjusted based on the branch.

Comment 27

[Jorg K (CEST = GMT+2)]

Which ones? D16933, D18133 and D24408? Of the four not yet published ones, those appear to apply to C-C. Please confirm.

Comment 28

[Tom Prince [:tomprince]]

(In reply to Jorg K (GMT+1) from comment #27)

Which ones? D16933, D18133 and D24408?

Yes, these. Thanks!

Comment 29

[Jorg K (CEST = GMT+2)]

https://hg.mozilla.org/comm-central/rev/a682f8ba00ff3feae6895e30269eac8b8261822b
https://hg.mozilla.org/comm-central/rev/aceb357e860520cc7ca73c0865347c4f032043d5
https://hg.mozilla.org/comm-central/rev/b491515368ac38db42e64d29b11329c5b5bb6525

Can someone explain to me why that pulsebot thing works one day but not the next :-(

Comment 30

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bb60a18552f3
1512188 - use new decision task image, with new options r=tomprince

Comment 31

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/2abbb5ff1172
[comm-task-env] Pass wrapped command name as argv[0]; r=dustin

Comment 32

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/bb60a18552f3
https://hg.mozilla.org/mozilla-central/rev/2abbb5ff1172

Comment 33

[Jorg K (CEST = GMT+2)]

Tom, you said in comment #26 to uplift the the C-C changes to C-B and C-ESR60. However, I see plenty of M-C landings on mozilla68 in comment #16 and comment #32. Don't these need uplift first? Rev 58dec282b0bc for example creates taskcluster/scripts/comm-task-env.

Comment 34

[Tom Prince [:tomprince]]

Jorg: The only thing those things affect is the docker image used by the decision task, which is built and pubblished by hand. The c-c changes point to the image built off of m-c, so don't depend on those change being in the tree being built.

Comment 35

[Jorg K (CEST = GMT+2)]

OK, I'll take it to the beta on my next round of uplifts. Can this be closed now?

Comment 36

[Tom Prince [:tomprince]]

and esr60?

Comment 37

[Jorg K (CEST = GMT+2)]

Sure, but we've just built 60.6.1. Are we in a hurry? Also repeating: Is the bug done?

Comment 38

[Jorg K (CEST = GMT+2)]

TB 67 beta:
https://hg.mozilla.org/releases/comm-beta/rev/e8b82a0fa34f0973d17c8ee990e5aa470d94d4d7
https://hg.mozilla.org/releases/comm-beta/rev/fe65720478f1bfed05751016e9a8661b270bd753
https://hg.mozilla.org/releases/comm-beta/rev/009e7920f8607c326e6d3c1c6f68f11c435911c4

I'll do the c-esr60 when TB 60.6.1 is out the door, so in the next 48 hours.

Looking at the patches, we have:
https://hg.mozilla.org/releases/comm-beta/rev/e8b82a0fa34f0973d17c8ee990e5aa470d94d4d7#l1.7

+GECKO_HEAD_REPOSITORY: https://hg.mozilla.org/releases/mozilla-beta

for beta and this for central:
https://hg.mozilla.org/comm-central/rev/a682f8ba00ff3feae6895e30269eac8b8261822b#l1.7

+GECKO_HEAD_REPOSITORY: https://hg.mozilla.org/mozilla-central

So when central merges to beta or beta merges to esr68 (yes, sixty-eight), it's not going to be right. Something in the merge scripts must fix this currently for .taskcluster.yml, so we have to adapt that something to the new regime.

Comment 39

[Patrick Cloke [:clokep]]

Yeah we have this scripted out and it should be easy enough to update for another file, I want to make sure I understand what change though, it looks like a new file was added (.gecko_rev.yml) where we need to update the GECKO_HEAD_REPOSITORY? And this configuration is no longer in .taskcluster.yml (so we no longer need to make changes there)? Is that correct?

Comment 40

[Jorg K (CEST = GMT+2)]

Exactly, as shown in comment #38. Also, when you first merge beta to esr, which is rare, you'd have to restore GECKO_HEAD_REF: default and remove any beta pinning from GECKO_HEAD_REV. I'll clear the NI for the other guys now.

Comment 41

[Jorg K (CEST = GMT+2)]

TB 60.6.2 or later:
https://hg.mozilla.org/releases/comm-esr60/rev/e0aa38099ad268fda5ed7f2381161e3df0fce4ed
https://hg.mozilla.org/releases/comm-esr60/rev/bcd99f0002987a443be14dcdd85026f69892d038
https://hg.mozilla.org/releases/comm-esr60/rev/8a8f9af60580591d3f22462a77af35cd152d811f

Tom told me that he wanted to leave this open until the last repository was updated to the new scheme. This is now the case (assuming that it worked).

Comment 42

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/comm-central/rev/1157a64ea65a
Follow-up: remove empty lines at the end of .gecko_rev.yml. rs=white-space-only

Comment 43

[Patrick Cloke [:clokep]]

Update our merge scripts today: https://hg.mozilla.org/users/bugzilla_standard8.plus.com/drivertools/rev/3dad4e1dbbdb4e2a7e055041f030cea2c4db4a7c