1397736 - Determine extra context information for cross-channel l10n commits

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Axel Hecht [:Pike]]

When creating commits, I add a bit of meta data to the commit's extra data.

gps, can you comment on the choice here:

convert_revision:
  40 hex string, original commit revision. Stolen from the convert extension
convert_url:
  URL to https://hg.mozilla.org/releases/mozilla-beta/rev/40_hex_string
x_channel_repo:
  "unified" name, so mozilla-central or comm-central, even for a commit coming from beta

The `convert_revision` is sweet, I think it'd only be a concern if we'd loose that meta data from an original commit. Doesn't sound likely?

`convert_url` I made up. I'm hoping to add that to the templates on hg.m.o eventually. We could either make this a full url, or just `releases/mozilla-beta` (named convert_repo_name?) and construct the url in the templates. Just using the full url seemed straightforward to me.

`x_channel_repo` is a helper, that allows me to skip all comm-central related commits if I'm looking for a good parent in the target repo for a mozilla-central related commit, effectively allowing me to linearize history a bit more.

Comment 1

[Gregory Szorc [:gps]]

Storing data in the changeset's "extra" fields isn't reliable. For example, some commands don't preserve all extra keys when rewriting changesets.

We instead prefer metadata be stored as "key: value" text in the commit message.

We have various fragments of code for performing the parsing and commit message manipulation. In v-c-t:

hgext/reviewboard/hgrb/util.py:addcommitid()
pylib/mozautomation/mozautomation/commitparser.py

IMO we should extract the "add a metadata line to commit message" bit to commitparser.py or to somewhere under pylib/mozhg.

Anyway, we already have a "convention" for identifying the source repo and revision:

  Source-Repo: <url>
  Source-Revision: <revision>

If you put these in commit messages, they will get auto hyperlinked on hg.mo.

Feel free to invent your own key for x-channel. Something like L10N-XChannel-Repo.

Comment 2

[Axel Hecht [:Pike]]

So, looking at https://hg.mozilla.org/mozilla-central/rev/6478b0b2e71c, that already has that information.

Can I just add another two of those at the end? Aka:

servo: Merge #18283 - User Timing API (from ferjm:user.timing.api); r=jdm

- [X] `./mach build -d` does not report any errors
- [X] `./mach test-tidy` does not report any errors
- [X] These changes fix #18109
- [X] There are tests for these changes. I enabled the peformance-timeline API WPTs but some of them are still failing because of implementation bugs or missing APIs (Resource Timing, for instance) the tests are dependent of. I'll file issues to fix them.

Source-Repo: https://github.com/servo/servo
Source-Revision: 1e93749941d2e3569c7c907832658c57ffb18c72

Source-Repo: https://hg.mozilla.org/mozilla-central
Source-Revision: .....


https://dxr.mozilla.org/hgcustom_version-control-tools/source/pylib/mozautomation/mozautomation/commitparser.py#285 doesn't look like it could do that multiple times.

Related, how performant and reliable is it to look up revisions based on that data? In https://hg.mozilla.org/users/axel_mozilla.com/cross-channel-experimental/file/482751cacd97/initial/__init__.py#l377, I'm looking up if the revision in m-c has been converted to something in the target repo.
That's reliable in the target repo I create up to hash collisions.
I could probably do both, one for reliable lookup in the repo I control, one for the case where people convert my repo to something.

Comment 3

[Axel Hecht [:Pike]]

Thinking about this over night, I think doing both is right. Public facing info in the message, and internal state-keeping in extra().

Caveats about Source-Revision: I've decided to remove it and Source-Repo if it's already in the commit message. I think that's fine, as folks can go to the original commit to see further redirects. Also, I think this is currently a hypothetical problem, as the only commits that do are servo commits, and they're not having strings, thus likely won't show in my history. We can revisit that later on, right now, the hg templates only linkify the first.
Apparently we're the first to do this with original revision being outside of github, so linkifying the revision doesn't work yet.

Anywho, I'm having the latest code doing a generation again, uploaded to https://hg.mozilla.org/users/axel_mozilla.com/en-x-testing/rev/9058a77af4f6, with

$ hg log -r 9058a77af4f6 -v
changeset:   9100:9058a77af4f6
user:        aceman <[e-mail]>
date:        Thu Aug 24 06:15:34 2017 +0200
files:       mail/chrome/messenger/addressbook/addressBook.properties suite/chrome/mailnews/addressbook/addressBook.properties
description:
Bug 46050 - Prevent duplicate address book names. r=jorgk, ui-r=thomasd,Paenglab

Source-Repo: https://hg.mozilla.org/comm-central
Source-Revision: aaf42ebb0c89b2b1e4d95fe8e395fe7b3d0df188


$ hg log -r 9058a77af4f6 -T'{join(extras,"\n")}\n'
branch=default
x_channel_revision_comm-central=aaf42ebb0c89b2b1e4d95fe8e395fe7b3d0df188

The idea is that we'll have the revisions of each unified repo in the extras, so once we have beta, it'd be something like

branch=default
x_channel_revision_comm-central=aaf42ebb0c89b2b1e4d95fe8e395fe7b3d0df188
x_channel_revision_releases/comm-beta=ee9edfba616b548d8c41e8ac06ad496fe198c745

The reason why I need both is that the commit in the target repo actually corresponds to the l10n state of central and beta at the same time. So when trying to find the latest change that corresponds to a new change on beta, I need to find this joined state. I prefixed this with x_channel_.
I went for a key per repo, 'cause extra() doesn't support data types beyond strings.

This way, the actual "public" reference is in the commit message, and the information in extra is only about my internal state keeping.
I think it's OK if tools that do something with the repo don't promise to keep that. It's really my shamap, and all I need from it is that the repo I push it to returns it back to me on cloning.

Comment 4

[Gregory Szorc [:gps]]

I'd prefer if we didn't have multiple metadata entries for the same key. Using something like Source-Repo-1 is better IMO.

I'll say it again: Mercurial's extra fields won't get preserved by certain Mercurial operations and may not get preserved by tools converting to/from Git. If you want the metadata to be reliable, put it in the commit message.

Comment 5

[Axel Hecht [:Pike]]

We ended up with 

X-Channel-Repo: mozilla-central
X-Channel-Converted-Revision: 0eaf0f3bce7a4effb16e8154c53b70feab0d46b5
X-Channel-Repo: releases/mozilla-beta
X-Channel-Revision: 3cdddb3af232d73c8c5c40f3dc083f65a54185c3

which works good in practice, and is commit-description-only, and doesn't need extra() at all.