Figure out how to pin thunderbird (comm-*) release branches that interacts nicely with actions as hooks.

RESOLVED FIXED in mozilla68

Status

task
RESOLVED FIXED
7 months ago
a month ago

People

(Reporter: tomprince, Assigned: rjl, NeedInfo)

Tracking

unspecified
mozilla68
Dependency tree / graph

Firefox Tracking Flags

(firefox68 affected)

Details

Attachments

(7 attachments)

(Reporter)

Description

7 months ago
Thunderbird is currently pinning revision of mozilla-* by adding `GECKO_HEAD_REV` to `.taskcluster.yml`. Since Bug 1415868 switched to using hooks for actions, the hooks encode a hash of .taskcluster.yml. That means that every time that a pin is updated, hooks need to be regenerated. This isn't scaleable.

Thus, we need to find a way to pin versions that doesn't involve changing `.taskcluster.yml`.
(Reporter)

Comment 1

7 months ago
Talking with :dustin, we came up with a few thoughts on how this could be done. They all involved creating a new file in comm-central that has the pinned version, but differed in what looks at that file.

1) Teach mozilla-taskcluster[1] about the new file, and have it populate the context with contents of it when processing .taskcluster.yml
2) Teach `run-task`[2] to look for the file before checking out repositories (behind a flag of some kind). There is a bit of a chicken and egg issues, in that comm-central gets checked out after mozilla-central, so the would need to query hg.mozilla.org directly for the file. 
3) Add a helper script to the decision-task image that wraps `run-task`, queries hg.mozilla.org and populates `GECKO_HEAD_REV`
before calling run-task.


[1] https://github.com/taskcluster/mozilla-taskcluster/
[2] https://searchfox.org/mozilla-central/source/taskcluster/scripts/run-task
(Reporter)

Comment 2

7 months ago
This should be resolved before Bug 1485680 gets uplifted to esr60, or merged to mozilla-beta.
mozilla-taskcluster will soon(?) be replaced with a hook, and hooks are not sufficiently expressive to support (1).

Hey Tom, are you still interested in this?

Flags: needinfo?(mozilla)
(Reporter)

Comment 5

3 months ago

Yes. This is going to become critical for Thunderbird, as the release promotion action is going to become a hook, and without this, the pinning that TB does will require releng intervention to create a new hook for many releases.

Flags: needinfo?(mozilla) → needinfo?(rob)
(Assignee)

Comment 6

3 months ago

Discussed in IRC, will be implementing something like:

<tomprince> rjl: My thinking was for a new script that is in the decision task image, that bascially does data = json.loads(urllib.get(os.environ['COMM_HEAD_REPOSITORY']/.../os.environ['COMM_HEAD_REV']/raw-file/pin-data.json); env = os.environ.copy(); env['GECKO_HEAD_REV'] = data['GECKO_HEAD_REV']; os.exec(sys.argv[1:])

Assignee: nobody → rob
Flags: needinfo?(rob)
(Assignee)

Comment 7

3 months ago
run-task-wrapper runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of the comm repository,
".gecko_ref.yml"

The current pinning method of updating ".taskcluster.yml" is no longer
supported.
(Assignee)

Comment 8

3 months ago

This is a WIP of the wrapper script for run-task. As suggested in comment 6, it will be added to the decision task image. The idea is that it will look at the COMM_HEAD_REPOSITORY and COMM_HEAD_REF environment variables that are set by Taskcluster and download a configuration file from that repository that contains the necessary information to set the GECKO_* environment variables that run-task needs.

In this revision, the downloaded file is named ".gecko_ref.yml" and will go in the root of the comm-* repositories. I opted to use YAML over JSON because I find it easier to work with.

defaults:
    gecko_base: mozilla-unified

comm-central:
    gecko_head: mozilla-central
    gecko_ref: default

comm-beta:
    gecko_head: mozilla-beta
    gecko_ref: abb8a4bb66a742bc1e31c3e2a2956bedf52be665

comm-esr60:
    gecko_head: mozilla-esr60
    gecko_ref: abb8a4bb66a742bc1e31c3e2a2956bedf52be665

try-comm-central:
    gecko_head: mozilla-central
    gecko_ref: default

It's written so a single file could be used for all of the comm repositories; That could be overkill. I've tried to keep it simple: just repository names and revisions.

The "command:" section of .taskcluster.yml gets a line added at the top:

        command:
          **- /builds/worker/bin/run-task-wrapper**
          - /builds/worker/bin/run-task
          - '--vcs-checkout=/builds/worker/checkouts/gecko'
          - '--sparse-profile=build/sparse-profiles/taskgraph'
          - '--comm-checkout=/builds/worker/checkouts/gecko/comm'
          - '--'
          - bash
          - -cx

The "GECKO_BASE_REPOSITORY", "GECKO_HEAD_REPOSITORY", and "GECKO_HEAD_REF" can get removed from .taskcluster.yml.
A new environment variable, "PROJECT_ENV_PREFIX" needs to be set. For Thunderbird repositories, it's just "COMM". This is used to construct the environment variable names "COMM_HEAD_REPOSITORY" and "COMM_HEAD_REF". The idea there being that there could be other "subprojects" that need this functionality.

I've only done some very basic testing locally.

(In reply to Rob Lemley [:rjl] from comment #8)

It's written so a single file could be used for all of the comm repositories; That could be overkill.

Wouldn't that make it hard to compare (i.e. diff) what needs to be done for the various versions?

(Assignee)

Comment 10

3 months ago

Modifying .taskcluster.yml to pin comm-* repositories to m-* repo/rev's is
not supported going forward. This file is the new canonical location for
those pinnings.

(Assignee)

Comment 11

3 months ago

run-task-wrapper runs before run-task and updates the environment with GECKO_*
variables that are defined in a file at the root of a subproject's repository,
such as "comm-central".

Updates:

  • add run-task-wrapper
  • add python 3.5 (run-task dependency)
  • add pyyaml
Attachment #9037453 - Attachment description: Bug 1491371 - Pin comm repositories without changing .taskcluster.yml → Bug 1491371 - Add config file for pinning to Mozilla repositories. r?tomprince r?jorgk
Attachment #9037453 - Attachment description: Bug 1491371 - Add config file for pinning to Mozilla repositories. r?tomprince r?jorgk → Bug 1491371 - Add config file for pinning to Mozilla repositories. r?tomprince
(Assignee)

Comment 12

3 months ago

Prepends/wraps "run-task" execution with "comm-run-task-wrapper". It will
download the configuration file containing the Gecko h.m.o. repository
information and set the appropriate environment variables so "run-task" can
run.

Attachment #9037130 - Attachment description: Bug 1491371 - Pin comm branches to mozilla revs without .taskcluster.yml. → Bug 1491371 - Pin comm branches to mozilla revs without .taskcluster.yml. r?dustin
Attachment #9037454 - Attachment description: Bug 1491371 - Update decision task image to support run-task-wrapper. → Bug 1491371 - Update decision task image to support comm-task-env. r?dustin
Attachment #9037453 - Attachment description: Bug 1491371 - Add config file for pinning to Mozilla repositories. r?tomprince → Bug 1491371 - Add config file for pinning to Mozilla repositories. r?dustin
Attachment #9040228 - Attachment description: Bug 1491371 - Update .taskcluster.yml to run comm-run-task-wrapper r?tomprince → Bug 1491371 - Update .taskcluster.yml to run comm-task-env r?dustin
Attachment #9037453 - Attachment description: Bug 1491371 - Add config file for pinning to Mozilla repositories. r?dustin → Bug 1491371 - Add config file for pinning to Mozilla repositories. r?darktrojan

Any timeframe for landing this? We're ready to land bug 1485680.

Flags: needinfo?(rob)
(Assignee)

Comment 14

a month ago

Per my conversation with Dustin earlier, this should be ready to go. I am personally not comfortable doing the decision task image update myself, so I've asked him to do that piece.

D16783 adds the wrapper script to m-c, then D16934 is the changes to Dockerfile/system-setup.sh to include that script.
I bumped the VERSION file at one point, as the docs said to do that early in the process.

Dustin, if you could build and deploy the new decision image incorporating those changes, I should be able to complete the rest as it's just the c-c changes. I'll be happy to coordinate timing with you. I am also unclear on what needs to be done with image hash updates that need to happen in various places.

Flags: needinfo?(rob) → needinfo?(dustin)

Comment 15

a month ago
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/integration/autoland/rev/58dec282b0bc
Pin comm branches to mozilla revs without .taskcluster.yml. r=tomprince,dustin
https://hg.mozilla.org/integration/autoland/rev/770808176c5c
Update decision task image to support comm-task-env. r=dustin

Comment 16

a month ago
bugherder
Status: NEW → RESOLVED
Last Resolved: a month ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla68
Status: RESOLVED → REOPENED
Flags: needinfo?(dustin)
Resolution: FIXED → ---

That second try push failed with

[taskcluster 2019-03-21 17:14:11.427Z] === Task Starting ===
[setup 2019-03-21T17:14:11.779Z] run-task started in /
usage: run-task [-h] [--user USER] [--group GROUP]
                [--gecko-checkout GECKO_CHECKOUT]
                [--gecko-sparse-profile GECKO_SPARSE_PROFILE]
                [--comm-checkout COMM_CHECKOUT]
                [--comm-sparse-profile COMM_SPARSE_PROFILE]
                [--fetch-hgfingerprint]
run-task: error: unrecognized arguments: --vcs-checkout=/builds/worker/checkouts/gecko --sparse-profile=build/sparse-profiles/taskgraph
[taskcluster 2019-03-21 17:14:12.025Z] === Task Finished ===

It looks like the work in bug 1512188 was only partly finished, and in particular .taskcluster.yml was never updated. I'll re-open to finish that.

Hm, I'll just solve it here actually.

Rob, you should be good to go (assuming that sticks and hits mozilla-central)

(Reporter)

Comment 26

a month ago

jorgk, can you get the comm patches landed and uplifted to all branches. I don't think there are any dependencies on m-c based patches that need to be uplifted. The .gecko_rev.yml will need to be adjusted based on the branch.

Flags: needinfo?(jorgk)

Comment 27

a month ago

Which ones? D16933, D18133 and D24408? Of the four not yet published ones, those appear to apply to C-C. Please confirm.

Flags: needinfo?(jorgk) → needinfo?(mozilla)
(Reporter)

Comment 28

a month ago

(In reply to Jorg K (GMT+1) from comment #27)

Which ones? D16933, D18133 and D24408?

Yes, these. Thanks!

Flags: needinfo?(mozilla)
(Reporter)

Updated

a month ago
Keywords: leave-open

Comment 30

a month ago
Pushed by dmitchell@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/bb60a18552f3
1512188 - use new decision task image, with new options r=tomprince

Comment 31

a month ago
Pushed by mozilla@hocat.ca:
https://hg.mozilla.org/integration/autoland/rev/2abbb5ff1172
[comm-task-env] Pass wrapped command name as argv[0]; r=dustin

Comment 33

a month ago

Tom, you said in comment #26 to uplift the the C-C changes to C-B and C-ESR60. However, I see plenty of M-C landings on mozilla68 in comment #16 and comment #32. Don't these need uplift first? Rev 58dec282b0bc for example creates taskcluster/scripts/comm-task-env.

Flags: needinfo?(mozilla)
(Reporter)

Comment 34

a month ago

Jorg: The only thing those things affect is the docker image used by the decision task, which is built and pubblished by hand. The c-c changes point to the image built off of m-c, so don't depend on those change being in the tree being built.

Flags: needinfo?(mozilla)

Comment 35

a month ago

OK, I'll take it to the beta on my next round of uplifts. Can this be closed now?

Updated

a month ago
Flags: needinfo?(jorgk)
(Reporter)

Comment 36

a month ago

and esr60?

Comment 37

a month ago

Sure, but we've just built 60.6.1. Are we in a hurry? Also repeating: Is the bug done?

Flags: needinfo?(jorgk)

Comment 38

a month ago

TB 67 beta:
https://hg.mozilla.org/releases/comm-beta/rev/e8b82a0fa34f0973d17c8ee990e5aa470d94d4d7
https://hg.mozilla.org/releases/comm-beta/rev/fe65720478f1bfed05751016e9a8661b270bd753
https://hg.mozilla.org/releases/comm-beta/rev/009e7920f8607c326e6d3c1c6f68f11c435911c4

I'll do the c-esr60 when TB 60.6.1 is out the door, so in the next 48 hours.

Looking at the patches, we have:
https://hg.mozilla.org/releases/comm-beta/rev/e8b82a0fa34f0973d17c8ee990e5aa470d94d4d7#l1.7

+GECKO_HEAD_REPOSITORY: https://hg.mozilla.org/releases/mozilla-beta

for beta and this for central:
https://hg.mozilla.org/comm-central/rev/a682f8ba00ff3feae6895e30269eac8b8261822b#l1.7

+GECKO_HEAD_REPOSITORY: https://hg.mozilla.org/mozilla-central

So when central merges to beta or beta merges to esr68 (yes, sixty-eight), it's not going to be right. Something in the merge scripts must fix this currently for .taskcluster.yml, so we have to adapt that something to the new regime.

Flags: needinfo?(rob)
Flags: needinfo?(mozilla)
Flags: needinfo?(clokep)

Yeah we have this scripted out and it should be easy enough to update for another file, I want to make sure I understand what change though, it looks like a new file was added (.gecko_rev.yml) where we need to update the GECKO_HEAD_REPOSITORY? And this configuration is no longer in .taskcluster.yml (so we no longer need to make changes there)? Is that correct?

Comment 40

a month ago

Exactly, as shown in comment #38. Also, when you first merge beta to esr, which is rare, you'd have to restore GECKO_HEAD_REF: default and remove any beta pinning from GECKO_HEAD_REV. I'll clear the NI for the other guys now.

Flags: needinfo?(rob)
Flags: needinfo?(mozilla)

Comment 41

a month ago

TB 60.6.2 or later:
https://hg.mozilla.org/releases/comm-esr60/rev/e0aa38099ad268fda5ed7f2381161e3df0fce4ed
https://hg.mozilla.org/releases/comm-esr60/rev/bcd99f0002987a443be14dcdd85026f69892d038
https://hg.mozilla.org/releases/comm-esr60/rev/8a8f9af60580591d3f22462a77af35cd152d811f

Tom told me that he wanted to leave this open until the last repository was updated to the new scheme. This is now the case (assuming that it worked).

Status: REOPENED → RESOLVED
Last Resolved: a month agoa month ago
Keywords: leave-open
Resolution: --- → FIXED

Comment 42

a month ago
Pushed by mozilla@jorgk.com:
https://hg.mozilla.org/comm-central/rev/1157a64ea65a
Follow-up: remove empty lines at the end of .gecko_rev.yml. rs=white-space-only
You need to log in before you can comment on or make changes to this bug.