Port bug 1615311 - merge day automation
Categories
(Thunderbird :: Build Config, enhancement)
Tracking
(thunderbird_esr78 unaffected)
Tracking | Status | |
---|---|---|
thunderbird_esr78 | --- | unaffected |
People
(Reporter: rjl, Assigned: rjl)
References
Details
Attachments
(7 files, 7 obsolete files)
1016 bytes,
patch
|
darktrojan
:
review+
|
Details | Diff | Splinter Review |
1.78 KB,
text/plain
|
Details | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
5.95 KB,
text/plain
|
Details | |
2.33 KB,
text/x-python
|
Details | |
47 bytes,
text/x-phabricator-request
|
Details | Review | |
47 bytes,
text/x-phabricator-request
|
Details | Review |
Initially this will just be a bustage fix, but I plan to implement for Thunderbird in the coming weeks.
Assignee | ||
Updated•4 years ago
|
Pushed by thunderbird@calypsoblue.org: https://hg.mozilla.org/comm-central/rev/0de98f3da93b Port bug 1615311L: Add merge-automation key to config.yml. rs=bustage-fix
Assignee | ||
Comment 2•4 years ago
|
||
This is what landed. Not sure where the L came from.
Assignee | ||
Updated•4 years ago
|
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 3•4 years ago
|
||
The above mentioned pull request against Treescript was accepted and merged today. I'll see what kind of config I can come up with for at least the central2beta merge. I would plan on using the current merge process on April 6 though.
Assignee | ||
Comment 4•4 years ago
|
||
Documenting what steps are run now and what needs to be migrated and what doesn't.
I will add this to the wiki page that we have as well.
https://wiki.mozilla.org/Thunderbird/Release_Driving/Rapid_Release_Activities/Merge_Repositories
Assignee | ||
Comment 5•4 years ago
|
||
Central2Beta - See the other attachment where I tried to map the existing script onto what Treescript can do. With the elimination of unnecessary steps and some minor code adjustments this should work.
Assignee | ||
Comment 6•4 years ago
|
||
Updated notes about shipped-locales files.
Assignee | ||
Comment 7•4 years ago
|
||
This doesn't work yet, but there's no danger in adding the configs. It does not run automatically.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 8•4 years ago
|
||
When testing this yesterday, I ran into a couple of problems.
- There's some new hardcoded paths within Treescript that won't work for Thunderbird. I've opened #206 for that. Just need to set up a way to point to a different path based on which project the merges are for.
- When testing against my test repos, I wasn't able to trigger the action. The error was a 403 Not Authorized. I looked at the hooks that are configured, and there is no hook corresponding to the merge-automation action for comm repositories. (They do exist for gecko). ci-admin generates those hooks.
I'm not sure why the hook isn't there. I'll note that the merge-automation kind config is not in comm-central yet, but I don't see where that's checked anywhere in ci-admin's in_tree_actions.py.
https://hg.mozilla.org/ci/ci-admin/file/9c9a898b351909b2e0fe8420ac9d649ded523af3/src/ciadmin/generate/in_tree_actions.py#l354
That looks like it could possibly have something to do with it? (ignore if the hook has never been fired)
NI? Tom for insight.
Comment 9•4 years ago
|
||
Comment on attachment 9145087 [details] [diff] [review] Merge day automation for comm-central to comm-beta Review of attachment 9145087 [details] [diff] [review]: ----------------------------------------------------------------- This looks okay to me, although my understanding of what it does is limited.
Assignee | ||
Comment 10•4 years ago
|
||
This covers the merge day activities that are necessary for the comm-central to comm-beta portion. Minor changes, adding 'fetch-version-from' and prefixing the behavior name as requested in D73722.
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 11•4 years ago
|
||
Comment on attachment 9145566 [details] [diff] [review] Merge day automation for comm-central to comm-beta Setting review status to match prior rev as the changes are minor.
Assignee | ||
Updated•4 years ago
|
Comment 12•4 years ago
|
||
Pushed by geoff@darktrojan.net:
https://hg.mozilla.org/comm-central/rev/029fd20ab19e
Merge day automation for comm-central to comm-beta. r=darktrojan
Updated•4 years ago
|
Comment 13•4 years ago
|
||
Pushed by geoff@darktrojan.net: https://hg.mozilla.org/comm-central/rev/7eb659891d33 follow-up - Fix YAML lint issues. rs=linting DONTBUILD
Assignee | ||
Updated•4 years ago
|
Comment 14•4 years ago
|
||
(In reply to Rob Lemley [:rjl] from comment #8)
- When testing against my test repos, I wasn't able to trigger the action. The error was a 403 Not Authorized. I looked at the hooks that are configured, and there is no hook corresponding to the merge-automation action for comm repositories. (They do exist for gecko). ci-admin generates those hooks.
The supported action permissions are defined in https://hg.mozilla.org/ci/ci-configuration/file/tip/actions.yml#l30
Assignee | ||
Updated•4 years ago
|
Assignee | ||
Comment 15•4 years ago
|
||
Assignee | ||
Comment 16•4 years ago
|
||
This should be everything needed for merge day activities except when the next esr gets created. I fixed a couple of mistakes from the last version.
Comment 17•4 years ago
|
||
Comment on attachment 9172798 [details] [diff] [review] 43314.patch Review of attachment 9172798 [details] [diff] [review]: ----------------------------------------------------------------- This looks fairly plausible, but I do have some questions which might be bogus! Does this make the proper changes to .gecko_rev.yml automatically? ::: taskcluster/ci/config.yml @@ +84,5 @@ > merge-automation: > + behaviors: > + comm-central-to-beta: > + fetch-version-from: "mail/config/version.txt" > + version-files: I'm guessing that this automatically strips the old suffix (a1)? @@ +91,5 @@ > + - filename: "mail/config/version_display.txt" > + new-suffix: 'b1' > + # Even though Seamonkey does not use Taskcluster, Thunderbird > + # developers do merge day activities, so we bump suite as well. > + - filename: "suite/config/version.txt" Is this going to overwrite Seamonkey's version with Firefoxes? @@ +95,5 @@ > + - filename: "suite/config/version.txt" > + new-suffix: '' > + - filename: "suite/config/version_display.txt" > + new-suffix: 'b1' > + replacements: We also remove `--enable-profiling` from each platform specific config. @@ +101,5 @@ > + - ac_add_options --with-branding=comm/mail/branding/nightly > + - ac_add_options --enable-official-branding > + merge-old-head: true > + base-tag: 'THUNDERBIRD_BETA_{major_version}_BASE' > + end-tag: 'THUNDERBIRD_BETA_{major_version}_END' Isn't the end-tag for the previous version? Or does this somehow take into account that happening? @@ +105,5 @@ > + end-tag: 'THUNDERBIRD_BETA_{major_version}_END' > + from-repo: 'https://hg.mozilla.org/comm-central' > + from-branch: 'comm' > + to-repo: 'https://hg.mozilla.org/releases/comm-beta' > + to-branch: 'comm-beta' What is from-branch/to-branch here? I expected these to just "default"?
Assignee | ||
Comment 18•4 years ago
|
||
(In reply to Patrick Cloke [:clokep] from comment #17)
Comment on attachment 9172798 [details] [diff] [review]
43314.patchReview of attachment 9172798 [details] [diff] [review]:
This looks fairly plausible, but I do have some questions which might be
bogus!Does this make the proper changes to .gecko_rev.yml automatically?
No, it doesn't. It could probably be added to Treescript though.
::: taskcluster/ci/config.yml
@@ +84,5 @@merge-automation:
- behaviors:
comm-central-to-beta:
fetch-version-from: "mail/config/version.txt"
version-files:
I'm guessing that this automatically strips the old suffix (a1)?
Right.
@@ +91,5 @@
- filename: "mail/config/version_display.txt"
new-suffix: 'b1'
# Even though Seamonkey does not use Taskcluster, Thunderbird
# developers do merge day activities, so we bump suite as well.
- filename: "suite/config/version.txt"
Is this going to overwrite Seamonkey's version with Firefoxes?
Oh right, they have different versioning. Maybe we leave this one out then. It's going to base the version on what it grabs from mail/config/version.txt so yes, it would be completely wrong for them.
@@ +95,5 @@
- filename: "suite/config/version.txt"
new-suffix: ''
- filename: "suite/config/version_display.txt"
new-suffix: 'b1'
replacements:
We also remove
--enable-profiling
from each platform specific config.
No reason to keep doing that. The nightly mozconfigs are not used for beta or esr builds.
@@ +101,5 @@
- ac_add_options --with-branding=comm/mail/branding/nightly
- ac_add_options --enable-official-branding
merge-old-head: true
base-tag: 'THUNDERBIRD_BETA_{major_version}_BASE'
end-tag: 'THUNDERBIRD_BETA_{major_version}_END'
Isn't the end-tag for the previous version? Or does this somehow take into
account that happening?
Yeah, Treescript does something different. It looks like it calculates the version for the end tag prior to updating. The logic baffles me right now though. https://github.com/mozilla-releng/scriptworker-scripts/blob/master/treescript/src/treescript/merges.py#L215-L219
@@ +105,5 @@
end-tag: 'THUNDERBIRD_BETA_{major_version}_END'
from-repo: 'https://hg.mozilla.org/comm-central'
from-branch: 'comm'
to-repo: 'https://hg.mozilla.org/releases/comm-beta'
to-branch: 'comm-beta'
What is from-branch/to-branch here? I expected these to just "default"?
Those are firefoxtree "branches". It works. Somehow.
We need to get a config commited to c-c so I can work out Taskcluster scope/permission issues with Aki. Then we can start doing test-runs against copies of the real repos in my users folder on hg.m.o. So we don't need to get this 100% correct right away, just enough for the next step.
Assignee | ||
Comment 19•4 years ago
|
||
Just removed the suite version file bump in this version since it would wind up being incorrect.
I'm not able to do dry-runs by just pushing to try-c-c due to missing scope on my account. I've been advised by releng to land the configuration in comm-central and see if the same problem is present.
Assignee | ||
Comment 20•4 years ago
|
||
I take it back...
Is this going to overwrite Seamonkey's version with Firefoxes?
Oh right, they have different versioning. Maybe we leave this one out then. It's going to base the version on what it grabs from mail/config/version.txt so yes, it would be completely wrong for them.
I just checked the Treescript code again. The version bump function updates the version files based on their contents. The tagging functions use "fetch_version_from" defined in config.yml. So the suite version files would not get clobbered with Firefox versions after all.
Except... The version bump code makes use of the mozilla_version Python module to validate what it's doing. It does not support Seamonkey versions. In this case the version class just needs to be able to parse the version and have a bump method. "Compatible enough".
For .gecko_rev.yml: Some work needs to be done, but I don't think it would be too bad. hg.m.o supports reading tags from a remote repo with URLs like: https://hg.mozilla.org/releases/mozilla-beta/json-tags. Basically, grab the to-repo tags, and take the newest one matching a regular expression.
Assignee | ||
Comment 21•4 years ago
|
||
This is a trimmed down log of a central-to-beta merge for Firefox. It can help clarify some of what goes on.
Assignee | ||
Updated•4 years ago
|
Comment 22•4 years ago
|
||
Ian -- not sure you're the right person, but the proposal in this bug is to automate some of the merge activities we do for Thunderbird, but seems incompatible with keeping the Seamonkey versions iterating at the current pace they're at. Would this be reasonable or is there good reason to bump those when we do merges?
Another option might be to bump the Seamonkey versions to be equivalent to the Thunderbird version, I have no idea if that's even remotely tractable though. (Also I have no idea if you're the right Seamonkey person, please redirect if not! :) )
Comment 23•4 years ago
|
||
(In reply to Patrick Cloke [:clokep] from comment #22)
Ian -- not sure you're the right person, but the proposal in this bug is to automate some of the merge activities we do for Thunderbird, but seems incompatible with keeping the Seamonkey versions iterating at the current pace they're at. Would this be reasonable or is there good reason to bump those when we do merges?
Another option might be to bump the Seamonkey versions to be equivalent to the Thunderbird version, I have no idea if that's even remotely tractable though. (Also I have no idea if you're the right Seamonkey person, please redirect if not! :) )
As far as I am aware, we still need the SeaMonkey versions bumping at the same pace/format as they are for the moment.
Would using version-bump: "minor" (similar to TB on comm-bump-esr) work on suite/config/version.txt (and version_display.txt)?
I would say on esr branch, if/when we need to bump the SeaMonkey version that can be done manually.
Comment 24•4 years ago
|
||
And if it works, tags would be of the format:
SEAMONKEY_BETA_{major_version}_{minor_version}_BASE
Assignee | ||
Comment 25•4 years ago
|
||
(In reply to Ian Neal from comment #24)
And if it works, tags would be of the format:
SEAMONKEY_BETA_{major_version}_{minor_version}_BASE
It would work to bump the Suite version files with "minor" just fine if we come up with some minimal support for Seamonkey versions in the mozilla_version Python module. (https://pypi.org/project/mozilla-version/). Or at least something compatible-enough that it can plug into Treescript.
Tag-wise, I can change the THUNDERBIRD part of the tag name to something more generic like COMM, but not a second set of tags with different version numbers like that. It is a change from our current BASE tags which use dates.
Comment 26•4 years ago
|
||
(In reply to Ian Neal from comment #23)
As far as I am aware, we still need the SeaMonkey versions bumping at the same pace/format as they are for the moment.
Would using version-bump: "minor" (similar to TB on comm-bump-esr) work on suite/config/version.txt (and version_display.txt)?
I would say on esr branch, if/when we need to bump the SeaMonkey version that can be done manually.
I actually had a thought last night after going offline that I wonder if we could bump the Seamonkey versions as a separate job / task that gets run after the merge? Not sure if that's possible. (And don't want to add a ton of effort onto Rob's plate.)
Assignee | ||
Comment 27•4 years ago
|
||
(In reply to Patrick Cloke [:clokep] from comment #26)
I actually had a thought last night after going offline that I wonder if we could bump the Seamonkey versions as a separate job / task that gets run after the merge? Not sure if that's possible. (And don't want to add a ton of effort onto Rob's plate.)
Either way Treescript doesn't know what to do with the Suite version.txt files. I wrote a SeamonkeyVersion class for mozilla-version last night. That package is maintained by Mozilla Releng so it should be possible to get it added.
It might be easier if the Suite bumps as a separate task. I'm not exactly sure how that would work right now though.
Patrick, can we at least get these changes as they are now landed? I can't do any dry runs to see what else might be broken until we do. We don't have to use it yet.
Assignee | ||
Comment 28•4 years ago
|
||
This code will be useful somewhere somehow.
Usage: pin_for_release.py mozilla-beta
json-tags returns the tags on a repo in a list with 0 being the most recent. So in the case of mozilla-beta or mozilla-esr78, when pinning for a release we would get the most recent RELEASE tag unless there are none and it finds BUILD1 first which would be the case for a Beta 1 pinning on merge day.
Probably. I'm sure there's bugs.
Assignee | ||
Comment 29•4 years ago
|
||
This version just drops THUNDERBIRD from the tag names to match what we have now.
BASE tag names are still changing from BETA_BASE_${date} to BETA_${major ver}_BASE. Personally I think that makes more sense anyway.
Assignee | ||
Comment 30•4 years ago
|
||
Is the comm-release repository used for anything? I looked through the push log going back to pre-68 and the only pushes I saw were the merge day ones, except for a lone push from Frank somewhere in there. Maybe it's a source repo for a mirror hosted elsewhere?
If comm-release is really unused, I'm inclined to retire it. When it comes time for the next esr we can skip the intermediary step and go right from comm-beta to comm-esr90.
Comment 31•4 years ago
|
||
Is the comm-release repository used for anything?
If retire means close it go for it but please do not remove. Use it regularly for cross referencing older stuff. SeaMonkey is currently basically using comm-central only. Everything else is maintained off site.
FRG
Assignee | ||
Comment 32•4 years ago
|
||
Yes by retire I mean 'stop updating on merge day'.
Assignee | ||
Comment 33•3 years ago
|
||
These additional grants allow thunderbird-releng to run merge automation on comm- repositories.
When testing merge automation on comm repositories, I get a missing scope error for:
hooks:trigger-hook:project-comm/in-tree-action-1-merge-automation/accbfbe031
On the Firefox side, I believe the permission is covered by the "- hooks:*" granted to
team_relops and releng at line 2511 of grants.yml.
Comment 34•3 years ago
|
||
Pushed by asasaki@mozilla.com: https://hg.mozilla.org/ci/ci-configuration/rev/071c776f97bd Allow Thunderbird releng to trigger merge-automation in-tree actions. r=releng-reviewers,aki
Assignee | ||
Comment 35•3 years ago
|
||
Progress!
Treescript now starts up, but then runs into a new error:
Running ['hg', 'up', '-C', 'comm-beta', '-R', '/app/workdir/src'] in /app ...
INFO - abort: unknown revision 'beta'!
ERROR - Failed to run async_main
Traceback (most recent call last):
File "/app/lib/python3.8/site-packages/scriptworker_client/client.py", line 182, in _handle_asyncio_loop
await async_main(config, task)
File "/app/lib/python3.8/site-packages/treescript/script.py", line 102, in async_main
await retry_async(do_actions, args=(config, task, actions_to_perform, repo_path), retry_exceptions=(CheckoutError, PushError))
File "/app/lib/python3.8/site-packages/scriptworker_client/aio.py", line 322, in retry_async
return await func(*args, **kwargs)
File "/app/lib/python3.8/site-packages/treescript/script.py", line 65, in do_actions
await perform_merge_actions(config, task, actions, repo_path, repo_type)
File "/app/lib/python3.8/site-packages/treescript/script.py", line 38, in perform_merge_actions
push_activity = await do_merge(config, task, repo_path)
File "/app/lib/python3.8/site-packages/treescript/merges.py", line 187, in do_merge
await run_hg_command(config, "up", "-C", to_branch, repo_path=repo_path)
File "/app/lib/python3.8/site-packages/treescript/mercurial.py", line 98, in run_hg_command
await run_command(command, env=env, exception=exception, log_path=log_path, **kwargs)
File "/app/lib/python3.8/site-packages/scriptworker_client/utils.py", line 244, in run_command
raise exception(
treescript.exceptions.FailedSubprocess: ('%s in %s exited %s!\n%s', ['hg', 'up', '-C', 'comm-beta', '-R', '/app/workdir/src'], '/app', 255, '')
exit code: 1
Further investigation is needed, but at least I have something that can be tested now.
Assignee | ||
Comment 36•3 years ago
|
||
Ahh of course...
The above error is coming from a very important difference between comm and mozilla repositories. There is no comm-unified repository that has all of the different "branches" in one place.
The steps taken are:
- hg robustcheckout the task source repo
- in my testing it happens to be try-c-c, in reality for central2beta it will be comm-central
- hg pull the "upstream_repo"
- for mozilla repos, this is mozilla-unified, and this is where all of the branch names get pulled in
- for comm repos, we always just use comm-central for upstream repo (as its required by robustcheckout) and it generally works
- hg up "to_branch" (in the run above, comm-beta)
- and this is where it falls apart for comm repos because the local copy does not know about comm-beta yet
- if treescript runs "hg pull comm-beta" prior to running "hg up comm-beta", it will work.
[update treescript locally]
After making some changes to Treescript to handle the fact that there is no comm-unified repository to use as an upstream, I was able to complete a comm-central to comm-beta merge. Still need to verify the end result is what's expected, but a quick glance at the result and the logs are promising.
Assignee | ||
Comment 37•3 years ago
|
||
In order to update suite/config/version.txt and version_display.txt, I've created a fork of mozilla-version with Seamonkey version number support at https://github.com/jfx2006/mozilla-version/tree/seamonkey_versions.
Before I send the pull request upstream, It would be helpful to know if what I came up with is "correct" or not. I don't know that all of the historical versions will parse correctly, but what I have should be enough for merge day work.
Comment 38•3 years ago
•
|
||
Rob,
for SeaMonkey you can simplfy it because we don't use taskcluster and only really care about updating and building comm-central right now. Everything else is in external repos until comm-central is fixed up which will still take time.
For merge day:
comm-central
patch level not used. Currently 2.81a1. Can go to 2.100a1 and up. Tested and works.
comm-central to comm comm-beta
Just cut a1
^(?P<major_number>\d+)
.(?P<minor_number>\d+)
comm-release is unused. If you still update it just copy comm-beta to comm-release and cut everything after the minor version.
For comm-beta to esr:
Initially same as for comm-release. No automatic updates after initial repo creation. Patch level will be set independently for releases.
^(?P<major_number>\d+)
.(?P<minor_number>\d+)
and cut everything after. Release versioning and adding patch levels will be done manually after the initial merge to esr.
Hope the description is clear. If not ewong or IanN shoculd correct me.
Dand thanks for doing this.
Assignee | ||
Comment 39•3 years ago
|
||
I'd like to use automation on next week's merge if possible.
I've just submitted pull request 309 for a Treescript update that fixes the problem I ran into in comment 36.
The one-repo project (bug 1666242) will make .gecko_rev.yml obsolete. With that in mind, I don't see any reason to implement automation for updating it unless one-repo completely flops.
The Suite version files will need to be updated in manually in a separate push for now. I can continue doing that until it's automated, or if the Seamonkey team prefers to do it that's fine as well. It makes sense to me that they are updated at the same time as the merges to keep things consistent, so I'll plan to add the functionality to Treescript in a followup.
One patch needs to land on comm-central, I'll upload that now for review.
Assignee | ||
Updated•3 years ago
|
Assignee | ||
Comment 40•3 years ago
|
||
Assignee | ||
Updated•3 years ago
|
Comment 41•3 years ago
|
||
Pushed by thunderbird@calypsoblue.org: https://hg.mozilla.org/comm-central/rev/0bb1b03c5fc8 Merge day automation configuration. r=justdave
Assignee | ||
Comment 42•3 years ago
|
||
Getting permission denied (public key)
from SSH with the stage user when testing a push to try-comm-central. There's a typo in the Treescript worker configuration with the ssh-merge user.
Running ['hg', 'push', '-e', 'ssh -l stage-ttbld-merge -i /app/configs/ssh_key_stage-ttbld-merge', '-r', 'a0a8f6506f68a0a1c2c9bc8368b29e0a87cee066', '-v', 'ssh://hg.mozilla.org/try-comm-central', '-R', '/app/workdir/src'] in /app ...
Aki - that should be stage-tbbld-merge.
Comment 43•3 years ago
|
||
Sorry about that. https://github.com/mozilla-services/cloudops-infra/pull/2938
Assignee | ||
Comment 44•3 years ago
|
||
Merge automation works!
Two caveats, which were known and art documented above.
- Suite versions are not done automatically yet, I did those in a second manual push.
- .gecko_rev.yml is not updated. I did that along with the suite versions. .gecko_rev.yml is going away hopefully before ESR91. I will add suite version support to TreeScript in a separate bug to be filed later.
Doc updates are in progress, for now in addition to the existing wiki there is http://docs.mozilla-releng.net/en/latest/procedures/release-duty/merge-duty/merge_duty.html and soon https://jfx2006.github.io/thunderbird-ci-docs/documentation/drivers/mergeautomation/
As we used to say... DUN DUN DUN!
And big thank you to Aki for his help!
Updated•3 years ago
|
Description
•