1973879 - Create a single orphan tags branch, and have it synced to all HgMO repos moving forwards

Status: RESOLVED FIXED

(Conduit :: git-to-hg sync, task, P2)

Description

[Olivier Mehani [:shtrom]]

1. confirm that tags to non-existent commits are not an issue on Hg
2. manually create the orphan branch (say, tags-unified)
   2.1. merge all tags branch into a temp branch
   2.2. copy the resulting .hgtags to this new branch
   2.3 push this branch to all repos (central, unified, beta, esr128, esr115, esr140, autoland)
3. update git-hg-sync config to only use this branch for all repos

Comment 1

[Julien Cristau [:jcristau] (back August 18)]

What's the point of 2.1 and 2.2?

Comment 2

[Olivier Mehani [:shtrom]]

That's to take all the existing tags from all the branches, and seed the .hgtags for the new branch.

This way all the tags should be in all the repos.

Comment 3

[Olivier Mehani [:shtrom]]

Which is why I need to confirm that it's ok, that Mercurial just ignores tags that points to commits it doesn't have, is step 1.

Comment 4

[Olivier Mehani [:shtrom]]

Note on the current state of git-hg-sync: the code assumes it has the source of truth (either in Git, or, in the case of tags, locally). It doesn't pull anything from HgMO, under the assumption that everything there has been push from the syncer.

When bootstrapping, this may not entirely the case. If the tags branch is not available locally, it is fetched from the Hg remote https://github.com/mozilla-conduit/git-hg-sync/blob/b5de079d8e637f87383e3d8d38dbba5e1bdcd180/git_hg_sync/repo_synchronizer.py#L136-L142. Moreover, the syncer only does this when processing a message for the target repository, which means the tag branch would get fetched from the target repo for the first message getting processed.

Moving to a single tags branch for all repo, as tags get synced back only to the target repo for the current message, this means that some of them may be lagging behind. This is not a problem for the repo themselves, but if their tags branch gets picked for the bootstrapping of the local syncer's branch,

EDIT: see bug 1962599

Comment 5

[Olivier Mehani [:shtrom]]

1. confirm that tags to non-existent commits are not an issue on Hg

Confirmed in dev as part of bug 1973880.

Comment 6

[Olivier Mehani [:shtrom]]

Attachment: [mozilla-conduit/git-hg-sync] config-prod: use tags-unified branch for all repos (bug 1973880, bug 1968531, bug 1973879) #62

Comment 7

[Olivier Mehani [:shtrom]]

With bug 1973880 having landed and been deployed, the remainder of this bug may become moot.

Comment 8

[Olivier Mehani [:shtrom]]

The config change was rolled back as to the new branch was hitting the singleroot hook. We'll allow-list this particular root in bug 1978262.

Comment 9

[Olivier Mehani [:shtrom]]

We'll have to re-add the tags that have been missed since the rollback, from https://github.com/mozilla-firefox/firefox/tags. Anything more recent than FIREFOX_141_0_BUILD2.

See https://bugzilla.mozilla.org/show_bug.cgi?id=1966014#c1 and https://bugzilla.mozilla.org/show_bug.cgi?id=1966014#c2 to generate fixing steps.

Comment 10

[Olivier Mehani [:shtrom]]

We can't really sort by tagging date (as the tags aren't annotated), but we get a rough order by committerdate, and filter by all commits at or after the commit tagged for FIREFOX_141_0_BUILD2 (

$ git -P tag --sort=-committerdate --format '%(refname:strip=2)%09%(objectname)%09%(committerdate)' | head -n 15
DEVEDITION_142_0b1_BUILD1       b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
DEVEDITION_142_0b1_RELEASE      b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
FIREFOX-ANDROID_142_0b1_BUILD1  b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
FIREFOX-ANDROID_142_0b1_RELEASE b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
FIREFOX_142_0b1_BUILD2  b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
FIREFOX_142_0b1_RELEASE b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6        Mon Jul 21 19:17:40 2025 +0000
FIREFOX_BETA_141_END    ff28c137f0ab5ee688319369fae667d5625f5b95        Mon Jul 21 09:04:57 2025 +0000
FIREFOX_BETA_142_BASE   55d2d902ca87120d28749315643b7339b68f9d1b        Mon Jul 21 05:47:24 2025 +0000
FIREFOX_NIGHTLY_142_END 55d2d902ca87120d28749315643b7339b68f9d1b        Mon Jul 21 05:47:24 2025 +0000
FIREFOX_141_0_BUILD2    c1ee0105d25a4c2b2887e916470bdf41a9fd47ef        Thu Jul 17 17:59:14 2025 +0000
FIREFOX_141_0_RELEASE   c1ee0105d25a4c2b2887e916470bdf41a9fd47ef        Thu Jul 17 17:59:14 2025 +0000
FIREFOX-ANDROID_141_0_BUILD1    2c34f5bf3f0df924e506b69c3ac854474b85e57e        Mon Jul 14 15:00:43 2025 +0000
FIREFOX-ANDROID_141_0_RELEASE   2c34f5bf3f0df924e506b69c3ac854474b85e57e        Mon Jul 14 15:00:43 2025 +0000
FIREFOX_141_0_BUILD1    2c34f5bf3f0df924e506b69c3ac854474b85e57e        Mon Jul 14 15:00:43 2025 +0000
FIREFOX_RELEASE_141_BASE        da536142961558cbbb1cdfd9771de5564b96dd92        Mon Jul 14 14:39:29 2025 +0000

(FIREFOX-ANDROID_141_0_BUILD1 and below are left here to show the beginning of earlier tags)

Comment 11

[Olivier Mehani [:shtrom]]

$ TAGS=$(git -P tag --sort=-committerdate --format '%(refname)'| sed /FIREFOX-ANDROID_141_0_BUILD1/Q)
$ git for-each-ref $TAGS  --format  'git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg %(objectname)) with %(refname:strip=2) a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified %(refname:strip=2) %(objectname)'  | bash
git cinnabar tag --force --message 'No bug - Tagging e57edfb82080a831a64d25ded292a8fd779f40e3 with DEVEDITION_139_0b7_RELEASE a=tagging CLOSED TREE DONTBUILD' --onto refs/heads/tags-beta DEVEDITION_139_0b7_RELEASE c0451d108b634261a89bc501068bf1212a256e4f
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with DEVEDITION_142_0b1_BUILD1 a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified DEVEDITION_142_0b1_BUILD1 b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with DEVEDITION_142_0b1_RELEASE a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified DEVEDITION_142_0b1_RELEASE b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with FIREFOX-ANDROID_142_0b1_BUILD1 a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX-ANDROID_142_0b1_BUILD1 b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with FIREFOX-ANDROID_142_0b1_RELEASE a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX-ANDROID_142_0b1_RELEASE b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg c1ee0105d25a4c2b2887e916470bdf41a9fd47ef) with FIREFOX_141_0_BUILD2 a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_141_0_BUILD2 c1ee0105d25a4c2b2887e916470bdf41a9fd47ef
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg c1ee0105d25a4c2b2887e916470bdf41a9fd47ef) with FIREFOX_141_0_RELEASE a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_141_0_RELEASE c1ee0105d25a4c2b2887e916470bdf41a9fd47ef
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with FIREFOX_142_0b1_BUILD2 a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_142_0b1_BUILD2 b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6) with FIREFOX_142_0b1_RELEASE a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_142_0b1_RELEASE b2ddfc5392e7e10f1e54a7b33253e9729d99aaf6
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg ff28c137f0ab5ee688319369fae667d5625f5b95) with FIREFOX_BETA_141_END a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_BETA_141_END ff28c137f0ab5ee688319369fae667d5625f5b95
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg 55d2d902ca87120d28749315643b7339b68f9d1b) with FIREFOX_BETA_142_BASE a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_BETA_142_BASE 55d2d902ca87120d28749315643b7339b68f9d1b
git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg 55d2d902ca87120d28749315643b7339b68f9d1b) with FIREFOX_NIGHTLY_142_END a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified FIREFOX_NIGHTLY_142_END 55d2d902ca87120d28749315643b7339b68f9d1b

Ready to be piped into a shell!

(We need --force to retag tags which already exist in some repos.)

Comment 12

[Olivier Mehani [:shtrom]]

Looking at https://hg-edge.mozilla.org/releases/mozilla-release/shortlog/tags-release, it seems we also want to include ANDROID_141_0_RELEASE, even though it's tagging an earlier commit. Same for FIREFOX_115_26_0esr_RELEASE, as per https://hg-edge.mozilla.org/releases/mozilla-esr115/shortlog/tags-esr115.

Looking further down the list of recent tags DEVEDITION_141_0b9_BUILD1 was the first that was done prior to the failing tags after the config for bug 1973880 was rolled back.

So,

$ TAGS=$(git -P tag --sort=-committerdate --format '%(refname)'| sed /DEVEDITION_141_0b9_BUILD1/Q)
$ git for-each-ref --sort=-committerdate $TAGS  --format  'git cinnabar tag --force --message "No bug - Tagging $(git cinnabar git2hg %(objectname)) with %(refname:strip=2) a=tagging CLOSED TREE DONTBUILD" --onto refs/heads/tags-unified %(refname:strip=2) %(objectname)'  | bash

Comment 13

[Olivier Mehani [:shtrom]]

Trying to push to conduit-testing/infra-testing triggered the (restored) singe_head_per_branch hook due to other heads already present in the repo. The hook itself doesn't differentiate between new and pre-existing heads.

I disabled it in the repo's .hg/hgrc to fix this.

Comment 14

[Olivier Mehani [:shtrom]]

Ok, :jcristau, I pushed the unified branch with the missing tags from last week to conduit-testing/infra-testing.

https://hg-edge.mozilla.org/conduit-testing/infra-testing/shortlog/tags-unified

There is an artifact in the .hgtags, in that each tag appears twice, e.g., https://hg-edge.mozilla.org/conduit-testing/infra-testing/rev/dd92da99eeb10d1b6df27b3fca1b167d5aae97e1

Glandium and I investigated (well, mostly Glandium, but I was there for the ride): this is due to the tag already being present on another branch, in which case cinnabar writes out the old tag value, before the new tag value. This happens even if that is the same reference, as is the case here. Mercurial has a similar behaviour.

As the purpose of this branch is to then be shared between all repos, we won't find ourselves in the situation where we have duplicate tags on multiple branches, so I don't think this will be an enduring issue moving forwards.

Let me know if you're OK with starting to push that branch to all the repos, and reenable the config to write tags only to that branch.

Comment 15

[BMO Github Automation]

Attachment: [mozilla-conduit/git-hg-sync] config-prod: restore use of tags-unified branch for all repos (bug 1973879) (#66)

Comment 16

[Olivier Mehani [:shtrom]]

Testing that we can push the tags-unified branch to multiple repos without issue.

I targeted the conduit-testing/ff-test-dev repo, which had the same multiple-head issue as in https://bugzilla.mozilla.org/show_bug.cgi?id=1973879#c13, so I disabled that hook, too. This problem keeps occuring because all our test repos are clones of mozilla-unified, which does have multiple heads on default, and that particular hook disabled

• https://hg-edge.mozilla.org/conduit-testing/infra-testing/log?rev=head()+and+branch(default)+and+not+closed()
• https://hg-edge.mozilla.org/conduit-testing/ff-test-dev/log?rev=head()+and+branch(default)+and+not+closed()

After disabling the hook, the branch was pushed and created correctly: https://hg-edge.mozilla.org/conduit-testing/ff-test-dev/shortlog/tags-unified

Comparatively, the prod repos that are to be targetted with this fix DO NOT have multiple OPEN heads on default, so I don't expect this to be an issue:

• https://hg-edge.mozilla.org/mozilla-unified/log?rev=head()+and+branch(default)+and+not+closed(), multiple heads but python:mozhghooks.single_head_per_branch.hook disabled
• https://hg-edge.mozilla.org/mozilla-central/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled
• https://hg-edge.mozilla.org/releases/mozilla-release/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled
• https://hg-edge.mozilla.org/releases/mozilla-beta/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled
• https://hg-edge.mozilla.org/releases/mozilla-esr115/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled
• https://hg-edge.mozilla.org/releases/mozilla-esr128/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled
• https://hg-edge.mozilla.org/releases/mozilla-esr140/log?rev=head()+and+branch(default)+and+not+closed(), python:mozhghooks.single_head_per_branch.hook enabled

Comment 17

[BMO Github Automation]

Authored by https://github.com/shtrom
https://github.com/mozilla-conduit/git-hg-sync/commit/bacb1ca7ae6b7a0294815bbc6119ed393c762dd9
[main] config-prod: restore use of tags-unified branch for all repos (bug 1973879)

Comment 18

[Julien Cristau [:jcristau] (back August 18)]

Looks ok to me. Thanks.