Closed Bug 1619767 Opened 1 year ago Closed 3 months ago

Port bug 1615311 - merge day automation

Categories

(Thunderbird :: Build Config, enhancement)

enhancement
Not set
normal

Tracking

(thunderbird_esr78 unaffected)

RESOLVED FIXED
Thunderbird 78.0
Tracking Status
thunderbird_esr78 --- unaffected

People

(Reporter: rjl, Assigned: rjl)

References

(Regressed 1 open bug)

Details

Attachments

(7 files, 7 obsolete files)

1016 bytes, patch
darktrojan
: review+
Details | Diff | Splinter Review
1.78 KB, text/plain
Details
47 bytes, text/x-phabricator-request
Details | Review
5.95 KB, text/plain
Details
2.33 KB, text/x-python
Details
47 bytes, text/x-phabricator-request
Details | Review
47 bytes, text/x-phabricator-request
Details | Review

Initially this will just be a bustage fix, but I plan to implement for Thunderbird in the coming weeks.

Keywords: leave-open
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/comm-central/rev/0de98f3da93b
Port bug 1615311L: Add merge-automation key to config.yml. rs=bustage-fix
This is what landed. Not sure where the L came from.
Attachment #9130584 - Flags: review?(geoff)
Assignee: nobody → rob
Status: NEW → ASSIGNED
Attachment #9130584 - Flags: review?(geoff) → review+

The above mentioned pull request against Treescript was accepted and merged today. I'll see what kind of config I can come up with for at least the central2beta merge. I would plan on using the current merge process on April 6 though.

Attached file Merge_Automation.md (obsolete) —

Documenting what steps are run now and what needs to be migrated and what doesn't.
I will add this to the wiki page that we have as well.
https://wiki.mozilla.org/Thunderbird/Release_Driving/Rapid_Release_Activities/Merge_Repositories

Central2Beta - See the other attachment where I tried to map the existing
script onto what Treescript can do. With the elimination of unnecessary
steps and some minor code adjustments this should work.
Depends on: 1632986
Attached file Merge_Automation.md

Updated notes about shipped-locales files.

Attachment #9140308 - Attachment is obsolete: true
This doesn't work yet, but there's no danger in adding the configs. It does
not run automatically.
Attachment #9145087 - Flags: review?(geoff)
Attachment #9140311 - Attachment is obsolete: true

When testing this yesterday, I ran into a couple of problems.

  • There's some new hardcoded paths within Treescript that won't work for Thunderbird. I've opened #206 for that. Just need to set up a way to point to a different path based on which project the merges are for.
  • When testing against my test repos, I wasn't able to trigger the action. The error was a 403 Not Authorized. I looked at the hooks that are configured, and there is no hook corresponding to the merge-automation action for comm repositories. (They do exist for gecko). ci-admin generates those hooks.

I'm not sure why the hook isn't there. I'll note that the merge-automation kind config is not in comm-central yet, but I don't see where that's checked anywhere in ci-admin's in_tree_actions.py.

https://hg.mozilla.org/ci/ci-admin/file/9c9a898b351909b2e0fe8420ac9d649ded523af3/src/ciadmin/generate/in_tree_actions.py#l354
That looks like it could possibly have something to do with it? (ignore if the hook has never been fired)

NI? Tom for insight.

Flags: needinfo?(mozilla)
Comment on attachment 9145087 [details] [diff] [review]
Merge day automation for comm-central to comm-beta

Review of attachment 9145087 [details] [diff] [review]:
-----------------------------------------------------------------

This looks okay to me, although my understanding of what it does is limited.
Attachment #9145087 - Flags: review?(geoff) → review+
Depends on: 1634985
This covers the merge day activities that are necessary for the comm-central
to comm-beta portion.

Minor changes, adding 'fetch-version-from' and prefixing the behavior name
as requested in D73722.
Attachment #9145087 - Attachment is obsolete: true
Depends on: 1635072
No longer depends on: 1634985
Comment on attachment 9145566 [details] [diff] [review]
Merge day automation for comm-central to comm-beta

Setting review status to match prior rev as the changes are minor.
Attachment #9145566 - Flags: review+

Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/029fd20ab19e
Merge day automation for comm-central to comm-beta. r=darktrojan

Status: ASSIGNED → RESOLVED
Closed: 1 year ago
Resolution: --- → FIXED
Target Milestone: --- → Thunderbird 78.0
Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/7eb659891d33
follow-up - Fix YAML lint issues. rs=linting DONTBUILD
Flags: needinfo?(mozilla)

(In reply to Rob Lemley [:rjl] from comment #8)

  • When testing against my test repos, I wasn't able to trigger the action. The error was a 403 Not Authorized. I looked at the hooks that are configured, and there is no hook corresponding to the merge-automation action for comm repositories. (They do exist for gecko). ci-admin generates those hooks.

The supported action permissions are defined in https://hg.mozilla.org/ci/ci-configuration/file/tip/actions.yml#l30

Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Regressions: 1637349
Regressions: 1643863
Attached patch 43314.patch (obsolete) — Splinter Review

This should be everything needed for merge day activities except when the next esr gets created. I fixed a couple of mistakes from the last version.

Attachment #9145566 - Attachment is obsolete: true
Attachment #9172798 - Flags: review?(clokep)
Comment on attachment 9172798 [details] [diff] [review]
43314.patch

Review of attachment 9172798 [details] [diff] [review]:
-----------------------------------------------------------------

This looks fairly plausible, but I do have some questions which might be bogus!

Does this make the proper changes to .gecko_rev.yml automatically?

::: taskcluster/ci/config.yml
@@ +84,5 @@
>  merge-automation:
> +    behaviors:
> +        comm-central-to-beta:
> +            fetch-version-from: "mail/config/version.txt"
> +            version-files:

I'm guessing that this automatically strips the old suffix (a1)?

@@ +91,5 @@
> +                - filename: "mail/config/version_display.txt"
> +                  new-suffix: 'b1'
> +                # Even though Seamonkey does not use Taskcluster, Thunderbird
> +                # developers do merge day activities, so we bump suite as well.
> +                - filename: "suite/config/version.txt"

Is this going to overwrite Seamonkey's version with Firefoxes?

@@ +95,5 @@
> +                - filename: "suite/config/version.txt"
> +                  new-suffix: ''
> +                - filename: "suite/config/version_display.txt"
> +                  new-suffix: 'b1'
> +            replacements:

We also remove `--enable-profiling` from each platform specific config.

@@ +101,5 @@
> +                  - ac_add_options --with-branding=comm/mail/branding/nightly
> +                  - ac_add_options --enable-official-branding
> +            merge-old-head: true
> +            base-tag: 'THUNDERBIRD_BETA_{major_version}_BASE'
> +            end-tag: 'THUNDERBIRD_BETA_{major_version}_END'

Isn't the end-tag for the previous version? Or does this somehow take into account that happening?

@@ +105,5 @@
> +            end-tag: 'THUNDERBIRD_BETA_{major_version}_END'
> +            from-repo: 'https://hg.mozilla.org/comm-central'
> +            from-branch: 'comm'
> +            to-repo: 'https://hg.mozilla.org/releases/comm-beta'
> +            to-branch: 'comm-beta'

What is from-branch/to-branch here? I expected these to just "default"?

(In reply to Patrick Cloke [:clokep] from comment #17)

Comment on attachment 9172798 [details] [diff] [review]
43314.patch

Review of attachment 9172798 [details] [diff] [review]:

This looks fairly plausible, but I do have some questions which might be
bogus!

Does this make the proper changes to .gecko_rev.yml automatically?

No, it doesn't. It could probably be added to Treescript though.

::: taskcluster/ci/config.yml
@@ +84,5 @@

merge-automation:

  • behaviors:
  •    comm-central-to-beta:
    
  •        fetch-version-from: "mail/config/version.txt"
    
  •        version-files:
    

I'm guessing that this automatically strips the old suffix (a1)?

Right.

@@ +91,5 @@

  •            - filename: "mail/config/version_display.txt"
    
  •              new-suffix: 'b1'
    
  •            # Even though Seamonkey does not use Taskcluster, Thunderbird
    
  •            # developers do merge day activities, so we bump suite as well.
    
  •            - filename: "suite/config/version.txt"
    

Is this going to overwrite Seamonkey's version with Firefoxes?

Oh right, they have different versioning. Maybe we leave this one out then. It's going to base the version on what it grabs from mail/config/version.txt so yes, it would be completely wrong for them.

@@ +95,5 @@

  •            - filename: "suite/config/version.txt"
    
  •              new-suffix: ''
    
  •            - filename: "suite/config/version_display.txt"
    
  •              new-suffix: 'b1'
    
  •        replacements:
    

We also remove --enable-profiling from each platform specific config.

No reason to keep doing that. The nightly mozconfigs are not used for beta or esr builds.

@@ +101,5 @@

  •              - ac_add_options --with-branding=comm/mail/branding/nightly
    
  •              - ac_add_options --enable-official-branding
    
  •        merge-old-head: true
    
  •        base-tag: 'THUNDERBIRD_BETA_{major_version}_BASE'
    
  •        end-tag: 'THUNDERBIRD_BETA_{major_version}_END'
    

Isn't the end-tag for the previous version? Or does this somehow take into
account that happening?

Yeah, Treescript does something different. It looks like it calculates the version for the end tag prior to updating. The logic baffles me right now though. https://github.com/mozilla-releng/scriptworker-scripts/blob/master/treescript/src/treescript/merges.py#L215-L219

@@ +105,5 @@

  •        end-tag: 'THUNDERBIRD_BETA_{major_version}_END'
    
  •        from-repo: 'https://hg.mozilla.org/comm-central'
    
  •        from-branch: 'comm'
    
  •        to-repo: 'https://hg.mozilla.org/releases/comm-beta'
    
  •        to-branch: 'comm-beta'
    

What is from-branch/to-branch here? I expected these to just "default"?

Those are firefoxtree "branches". It works. Somehow.

We need to get a config commited to c-c so I can work out Taskcluster scope/permission issues with Aki. Then we can start doing test-runs against copies of the real repos in my users folder on hg.m.o. So we don't need to get this 100% correct right away, just enough for the next step.

Just removed the suite version file bump in this version since it would wind up being incorrect.

I'm not able to do dry-runs by just pushing to try-c-c due to missing scope on my account. I've been advised by releng to land the configuration in comm-central and see if the same problem is present.

Attachment #9172798 - Attachment is obsolete: true
Attachment #9172798 - Flags: review?(clokep)
Attachment #9172948 - Flags: review?(clokep)

I take it back...

Is this going to overwrite Seamonkey's version with Firefoxes?
Oh right, they have different versioning. Maybe we leave this one out then. It's going to base the version on what it grabs from mail/config/version.txt so yes, it would be completely wrong for them.

I just checked the Treescript code again. The version bump function updates the version files based on their contents. The tagging functions use "fetch_version_from" defined in config.yml. So the suite version files would not get clobbered with Firefox versions after all.
Except... The version bump code makes use of the mozilla_version Python module to validate what it's doing. It does not support Seamonkey versions. In this case the version class just needs to be able to parse the version and have a bump method. "Compatible enough".

For .gecko_rev.yml: Some work needs to be done, but I don't think it would be too bad. hg.m.o supports reading tags from a remote repo with URLs like: https://hg.mozilla.org/releases/mozilla-beta/json-tags. Basically, grab the to-repo tags, and take the newest one matching a regular expression.

Attached file merge_log.txt

This is a trimmed down log of a central-to-beta merge for Firefox. It can help clarify some of what goes on.

Ian -- not sure you're the right person, but the proposal in this bug is to automate some of the merge activities we do for Thunderbird, but seems incompatible with keeping the Seamonkey versions iterating at the current pace they're at. Would this be reasonable or is there good reason to bump those when we do merges?

Another option might be to bump the Seamonkey versions to be equivalent to the Thunderbird version, I have no idea if that's even remotely tractable though. (Also I have no idea if you're the right Seamonkey person, please redirect if not! :) )

Flags: needinfo?(iann_bugzilla)

(In reply to Patrick Cloke [:clokep] from comment #22)

Ian -- not sure you're the right person, but the proposal in this bug is to automate some of the merge activities we do for Thunderbird, but seems incompatible with keeping the Seamonkey versions iterating at the current pace they're at. Would this be reasonable or is there good reason to bump those when we do merges?

Another option might be to bump the Seamonkey versions to be equivalent to the Thunderbird version, I have no idea if that's even remotely tractable though. (Also I have no idea if you're the right Seamonkey person, please redirect if not! :) )

As far as I am aware, we still need the SeaMonkey versions bumping at the same pace/format as they are for the moment.
Would using version-bump: "minor" (similar to TB on comm-bump-esr) work on suite/config/version.txt (and version_display.txt)?
I would say on esr branch, if/when we need to bump the SeaMonkey version that can be done manually.

Flags: needinfo?(iann_bugzilla) → needinfo?(clokep)

And if it works, tags would be of the format:
SEAMONKEY_BETA_{major_version}_{minor_version}_BASE

(In reply to Ian Neal from comment #24)

And if it works, tags would be of the format:
SEAMONKEY_BETA_{major_version}_{minor_version}_BASE

It would work to bump the Suite version files with "minor" just fine if we come up with some minimal support for Seamonkey versions in the mozilla_version Python module. (https://pypi.org/project/mozilla-version/). Or at least something compatible-enough that it can plug into Treescript.

Tag-wise, I can change the THUNDERBIRD part of the tag name to something more generic like COMM, but not a second set of tags with different version numbers like that. It is a change from our current BASE tags which use dates.

(In reply to Ian Neal from comment #23)

As far as I am aware, we still need the SeaMonkey versions bumping at the same pace/format as they are for the moment.
Would using version-bump: "minor" (similar to TB on comm-bump-esr) work on suite/config/version.txt (and version_display.txt)?
I would say on esr branch, if/when we need to bump the SeaMonkey version that can be done manually.

I actually had a thought last night after going offline that I wonder if we could bump the Seamonkey versions as a separate job / task that gets run after the merge? Not sure if that's possible. (And don't want to add a ton of effort onto Rob's plate.)

Flags: needinfo?(clokep)

(In reply to Patrick Cloke [:clokep] from comment #26)

I actually had a thought last night after going offline that I wonder if we could bump the Seamonkey versions as a separate job / task that gets run after the merge? Not sure if that's possible. (And don't want to add a ton of effort onto Rob's plate.)

Either way Treescript doesn't know what to do with the Suite version.txt files. I wrote a SeamonkeyVersion class for mozilla-version last night. That package is maintained by Mozilla Releng so it should be possible to get it added.
It might be easier if the Suite bumps as a separate task. I'm not exactly sure how that would work right now though.

Patrick, can we at least get these changes as they are now landed? I can't do any dry runs to see what else might be broken until we do. We don't have to use it yet.

Attached file pin_for_release.py

This code will be useful somewhere somehow.
Usage: pin_for_release.py mozilla-beta

json-tags returns the tags on a repo in a list with 0 being the most recent. So in the case of mozilla-beta or mozilla-esr78, when pinning for a release we would get the most recent RELEASE tag unless there are none and it finds BUILD1 first which would be the case for a Beta 1 pinning on merge day.
Probably. I'm sure there's bugs.

This version just drops THUNDERBIRD from the tag names to match what we have now.
BASE tag names are still changing from BETA_BASE_${date} to BETA_${major ver}_BASE. Personally I think that makes more sense anyway.

Attachment #9172948 - Attachment is obsolete: true
Attachment #9172948 - Flags: review?(clokep)

Is the comm-release repository used for anything? I looked through the push log going back to pre-68 and the only pushes I saw were the merge day ones, except for a lone push from Frank somewhere in there. Maybe it's a source repo for a mirror hosted elsewhere?

If comm-release is really unused, I'm inclined to retire it. When it comes time for the next esr we can skip the intermediary step and go right from comm-beta to comm-esr90.

Flags: needinfo?(iann_bugzilla)

Is the comm-release repository used for anything?

If retire means close it go for it but please do not remove. Use it regularly for cross referencing older stuff. SeaMonkey is currently basically using comm-central only. Everything else is maintained off site.

FRG

Flags: needinfo?(iann_bugzilla)

Yes by retire I mean 'stop updating on merge day'.

These additional grants allow thunderbird-releng to run merge automation on comm- repositories.

When testing merge automation on comm repositories, I get a missing scope error for:
hooks:trigger-hook:project-comm/in-tree-action-1-merge-automation/accbfbe031

On the Firefox side, I believe the permission is covered by the "- hooks:*" granted to
team_relops and releng at line 2511 of grants.yml.

Pushed by asasaki@mozilla.com:
https://hg.mozilla.org/ci/ci-configuration/rev/071c776f97bd
Allow Thunderbird releng to trigger merge-automation in-tree actions. r=releng-reviewers,aki

Progress!
Treescript now starts up, but then runs into a new error:

Running ['hg', 'up', '-C', 'comm-beta', '-R', '/app/workdir/src'] in /app ...
INFO - abort: unknown revision 'beta'!
ERROR - Failed to run async_main
Traceback (most recent call last):
  File "/app/lib/python3.8/site-packages/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
    await async_main(config, task)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 102, in async_main
    await retry_async(do_actions, args=(config, task, actions_to_perform, repo_path), retry_exceptions=(CheckoutError, PushError))
  File "/app/lib/python3.8/site-packages/scriptworker_client/aio.py", line 322, in retry_async
    return await func(*args, **kwargs)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 65, in do_actions
    await perform_merge_actions(config, task, actions, repo_path, repo_type)
  File "/app/lib/python3.8/site-packages/treescript/script.py", line 38, in perform_merge_actions
    push_activity = await do_merge(config, task, repo_path)
  File "/app/lib/python3.8/site-packages/treescript/merges.py", line 187, in do_merge
    await run_hg_command(config, "up", "-C", to_branch, repo_path=repo_path)
  File "/app/lib/python3.8/site-packages/treescript/mercurial.py", line 98, in run_hg_command
    await run_command(command, env=env, exception=exception, log_path=log_path, **kwargs)
  File "/app/lib/python3.8/site-packages/scriptworker_client/utils.py", line 244, in run_command
    raise exception(
treescript.exceptions.FailedSubprocess: ('%s in %s exited %s!\n%s', ['hg', 'up', '-C', 'comm-beta', '-R', '/app/workdir/src'], '/app', 255, '')
exit code: 1

Further investigation is needed, but at least I have something that can be tested now.

Ahh of course...

The above error is coming from a very important difference between comm and mozilla repositories. There is no comm-unified repository that has all of the different "branches" in one place.

The steps taken are:

  • hg robustcheckout the task source repo
    • in my testing it happens to be try-c-c, in reality for central2beta it will be comm-central
  • hg pull the "upstream_repo"
    • for mozilla repos, this is mozilla-unified, and this is where all of the branch names get pulled in
    • for comm repos, we always just use comm-central for upstream repo (as its required by robustcheckout) and it generally works
  • hg up "to_branch" (in the run above, comm-beta)
    • and this is where it falls apart for comm repos because the local copy does not know about comm-beta yet
    • if treescript runs "hg pull comm-beta" prior to running "hg up comm-beta", it will work.

[update treescript locally]

After making some changes to Treescript to handle the fact that there is no comm-unified repository to use as an upstream, I was able to complete a comm-central to comm-beta merge. Still need to verify the end result is what's expected, but a quick glance at the result and the logs are promising.

In order to update suite/config/version.txt and version_display.txt, I've created a fork of mozilla-version with Seamonkey version number support at https://github.com/jfx2006/mozilla-version/tree/seamonkey_versions.
Before I send the pull request upstream, It would be helpful to know if what I came up with is "correct" or not. I don't know that all of the historical versions will parse correctly, but what I have should be enough for merge day work.

Flags: needinfo?(frgrahl)

Rob,
for SeaMonkey you can simplfy it because we don't use taskcluster and only really care about updating and building comm-central right now. Everything else is in external repos until comm-central is fixed up which will still take time.

For merge day:
comm-central
patch level not used. Currently 2.81a1. Can go to 2.100a1 and up. Tested and works.

comm-central to comm comm-beta
Just cut a1

^(?P<major_number>\d+)
.(?P<minor_number>\d+)

comm-release is unused. If you still update it just copy comm-beta to comm-release and cut everything after the minor version.

For comm-beta to esr:

Initially same as for comm-release. No automatic updates after initial repo creation. Patch level will be set independently for releases.

^(?P<major_number>\d+)
.(?P<minor_number>\d+)

and cut everything after. Release versioning and adding patch levels will be done manually after the initial merge to esr.

Hope the description is clear. If not ewong or IanN shoculd correct me.

Dand thanks for doing this.

Flags: needinfo?(frgrahl)

I'd like to use automation on next week's merge if possible.

I've just submitted pull request 309 for a Treescript update that fixes the problem I ran into in comment 36.

The one-repo project (bug 1666242) will make .gecko_rev.yml obsolete. With that in mind, I don't see any reason to implement automation for updating it unless one-repo completely flops.

The Suite version files will need to be updated in manually in a separate push for now. I can continue doing that until it's automated, or if the Seamonkey team prefers to do it that's fine as well. It makes sense to me that they are updated at the same time as the merges to keep things consistent, so I'll plan to add the functionality to Treescript in a followup.

One patch needs to land on comm-central, I'll upload that now for review.

Attachment #9181831 - Attachment is obsolete: true
Keywords: leave-open
Pushed by thunderbird@calypsoblue.org:
https://hg.mozilla.org/comm-central/rev/0bb1b03c5fc8
Merge day automation configuration. r=justdave
Depends on: 1682059

Getting permission denied (public key) from SSH with the stage user when testing a push to try-comm-central. There's a typo in the Treescript worker configuration with the ssh-merge user.

Running ['hg', 'push', '-e', 'ssh -l stage-ttbld-merge -i /app/configs/ssh_key_stage-ttbld-merge', '-r', 'a0a8f6506f68a0a1c2c9bc8368b29e0a87cee066', '-v', 'ssh://hg.mozilla.org/try-comm-central', '-R', '/app/workdir/src'] in /app ...

Aki - that should be stage-tbbld-merge.

Flags: needinfo?(aki)

Merge automation works!

Two caveats, which were known and art documented above.

  1. Suite versions are not done automatically yet, I did those in a second manual push.
  2. .gecko_rev.yml is not updated. I did that along with the suite versions. .gecko_rev.yml is going away hopefully before ESR91. I will add suite version support to TreeScript in a separate bug to be filed later.

Doc updates are in progress, for now in addition to the existing wiki there is http://docs.mozilla-releng.net/en/latest/procedures/release-duty/merge-duty/merge_duty.html and soon https://jfx2006.github.io/thunderbird-ci-docs/documentation/drivers/mergeautomation/

As we used to say... DUN DUN DUN!

And big thank you to Aki for his help!

Status: REOPENED → RESOLVED
Closed: 1 year ago3 months ago
Resolution: --- → FIXED
You need to log in before you can comment on or make changes to this bug.