1035934 - Named branches not syncing properly in legacy vcs-sync

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Hal Wine [:hwine] use NI!]

(In reply to Henrik Skupin (:whimboo) from bug 799719 comment #9)
> When we can keep the history for open branches, and can get rid of all the
> closed ones by re-creating the repo, I'm fine.
> 
> A bigger problem is currently the mirroring of named branches, which does
> not happen at all! What I can see only the default branch is mirrored and
> that's all. 
> 
> https://github.com/mozilla/qa-mozmill-tests/tree/master (6 days)
> https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-aurora (2 months)
> https://github.com/mozilla/qa-mozmill-tests/tree/mozilla-beta (2 months)
> 
> The 2 months is exactly this time when we got the mirroring setup.
> 
> So please make sure that all named branches are getting mirrored.

Moving this separate issue to a separate bug.

Comment 1

[Hal Wine [:hwine] use NI!]

Definite bug -- will manifest on all hg repositories with multiple named branches being converted to git. Fortunately, that's not very many used directly by devs. (And specifically excludes gecko.git which is handled differently.)

The problem is:
 a) hg doesn't allow a branch and bookmark by same name
 b) hggit only converts bookmarks to git branches
 c) (b) means we have to force (-f) the names to be equal
 d) when resolving names to revisions, bookmarks have higher priority than branches

Our conversion code must update the bookmark to the latest branch point prior to conversion. Condition (d) says, after the first time, we always update the bookmark to the bookmark (i.e. no change), as the bookmarking code uses (the same) names. (This does not happen for the default hg branch, as that is mapped to the bookmark 'master', in line with git conventions.)

Fix would be to parse the branch revision, and thus be able to set the bookmark to a numeric revision. This would change the behavior on the FFOS l10n repositories, which are already partner visible (56 repos). The change is minimal, as the branches are already created, and few, if any, string changes have been made on those release branches.

Comment 2

[Hal Wine [:hwine] use NI!]

Attachment: bookmark_name.patch

tested on copy of conversion repository with following results:
before first run:
    $ ( cd mozilla-qa-mozmill-tests/; hg branches ; hg bookmarks; ) | grep mozilla-aurora
    mozilla-aurora              3957:daddd5c45709
    mozilla-aurora            3874:4e2937c6dcb4

after run of unmodified script (no change):
    $ ( cd mozilla-qa-mozmill-tests/; hg branches ; hg bookmarks; ) | grep mozilla-aurora
    mozilla-aurora              3957:daddd5c45709
    mozilla-aurora            3874:4e2937c6dcb4

after run with modified script - bookmark updated to branch rev:
    $ ( cd mozilla-qa-mozmill-tests/; hg branches ; hg bookmarks; ) | grep mozilla-aurora
    mozilla-aurora              3957:daddd5c45709
    mozilla-aurora            3957:daddd5c45709

debug output showing correct parsing of lines on closed branch:
    + read line
    + echo ''\''mozilla-esr17               3670:74171c915bcc (closed)'\'''
    'mozilla-esr17               3670:74171c915bcc (closed)'
    + branch=mozilla-esr17
    + branch_hash='74171c915bcc (closed)'
    + branch_hash=74171c915bcc
    + run_cmd hg --cwd mozilla-qa-mozmill-tests bookmark -f -r 74171c915bcc mozilla-esr17

open branch:
    + echo ''\''mozilla-esr24               3908:639ee4bb4ea0'\'''
    'mozilla-esr24               3908:639ee4bb4ea0'
    + branch=mozilla-esr24
    + branch_hash=639ee4bb4ea0
    + branch_hash=639ee4bb4ea0
    + run_cmd hg --cwd mozilla-qa-mozmill-tests bookmark -f -r 639ee4bb4ea0 mozilla-esr24

Comment 3

[Hal Wine [:hwine] use NI!]

(In reply to Hal Wine [:hwine] (use needinfo) from comment #1)
> ... This would change the behavior on the FFOS
> l10n repositories, which are already partner visible (56 repos). The change
> is minimal, as the branches are already created, and few, if any, string
> changes have been made on those release branches.

I discussed with Aki in IRC, and we both agree that no partner impacting change is involved.

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8453396 [details] [diff] [review]
bookmark_name.patch

This looks good in general.
What were you worried about on the l10n repos? that the branches would change which sha they point at?  If so, it might be worth finding a potentially problematic repo and verifying what happens on the conversion (but not pushing).

Comment 5

[Hal Wine [:hwine] use NI!]

(In reply to Aki Sasaki [:aki] from comment #4)
> What were you worried about on the l10n repos? that the branches would
> change which sha they point at?

If the sha changes, it will be to correct things :) So, concern was not about the change being made, but about any change causing confusion for partners.

The l10n repos are the only repositories (outside of qa-mozmill-tests) that both:
 - have hg native branches
 - are converted directly & entirely to git (gecko.git is selective)

That means those are the only other repositories that will be potentially affected by this fix. To be affected, a given repository would also need to have all of:
 - an hg branch already converted by the legacy system
 - a subsequent push with commits on the already-converted branch

If there were any such commits, they have been converted to git (thus present in mapper), but not yet pushed to git.m.o. That wouldn't break any of our build process, but a partner wouldn't be able to recreate exactly. Since we haven't had reports of this, it's unlikely it has happened.

I'll monitor the logs for the first-conversion-after-fix-deployed for any sign of changes, and notify partners if needed.

Comment 6

[Hal Wine [:hwine] use NI!]

Comment on attachment 8453396 [details] [diff] [review]
bookmark_name.patch

landed:
 http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/rev/0ea8a87b7332

deployed to all prodcution boxes

will close after first l10n run results in

Comment 7

[Hal Wine [:hwine] use NI!]

Attachment: bookmark_bustage.patch

Previous change removed line:
-        for x in $(hg --cwd $repo branches -c | grep -v ^default | cut -f1 -d " "); do

However, the replacement code did not include the 'grep -v ^default' functionality. This patch adds that back in.

_lots_ of cleanup to do afterwards, as all git repos now have an extra 'default' branch they shouldn't (except, thankfully, for gecko.git). Cleanup will be in separate bug.

Comment 8

[Hal Wine [:hwine] use NI!]

Comment on attachment 8454024 [details] [diff] [review]
bookmark_bustage.patch

landed:
 http://hg.mozilla.org/users/hwine_mozilla.com/repo-sync-tools/rev/a11fd9328659

in production

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Looks like it is working fine (beside the default branch issue) across all of our branches in the qa-mozmill-tests repository. Thanks Hal!

Comment 10

[Hal Wine [:hwine] use NI!]

Great -- thanks for reporting the issue!