1331697 - Mercurial extension to synchronize changesets between subdirectories in different repos

Status: RESOLVED FIXED

(Developer Services :: General, task)

Description

[Gregory Szorc [:gps]]

The goal of this bug is to write Mercurial commands that make it easy to manage a monorepo with projects in sub-directories by providing a mechanism to import and export changes within those sub-directories from and to repositories having only the history of this sub-directories. Think of it as a poor man's "narrow clone." The underlying goal is to support project/sub-directory-isolated development in Git.

There are 2 scenarios we want to support:

1) Replay changesets from a source repository into a sub-directory of a destination repository

2) Replay changesets from a sub-directory of a source repository into a destination repository

I have code for #1 mostly written. Just needs more tests and some polish. I may or may not get to #2 in this bug, as it isn't in the critical path for bug 1322769.

Comment 1

[Gregory Szorc [:gps]]

Attachment: overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

TODO:

* Write commit message and supporting docs
* Support multiple source repos in destination (need extras metadata,
  stable URL for source repo)
* Figure out mitigation for spoofing of extras metadata

Review commit: https://reviewboard.mozilla.org/r/105688/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/105688/

Comment 2

[Gregory Szorc [:gps]]

FWIW, I started to refactor the code to follow the model of `git subtree`, which is basically what I aim to eventually build. However, I realized after about an hour or so that it was going to be a significant time sink. I didn't want to delay the servo syncing any more. So for now we have a standalone `hg overlay` command as opposed to say `hg subtreepull`.

Comment 3

[Gregory Szorc [:gps]]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/105688/diff/1-2/

Comment 4

[:glob ✱]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

https://reviewboard.mozilla.org/r/105688/#review109532

this looks great as always.

::: hgext/overlay/__init__.py:45
(Diff revision 2)
> +    assert prefix.endswith('/')
> +
> +    sourceman = sourcectx.manifest()
> +    destman = destctx.manifest()
> +
> +    sourcefiles = set(sourceman.iterkeys())

do we care about python3 support here?
iterkeys() won't work on py3.

::: hgext/overlay/__init__.py:269
(Diff revision 2)
> +    """Mirror a source repository into the .hg directory of another."""
> +    u = util.url(url)
> +    if u.islocal():
> +        raise error.Abort(_('source repo cannot be local'))
> +
> +    path = store.encodefilename(url.replace('://', '_')).replace('/', '_')

the protocol should probably be excluded from the path; switching protocols for the same repo shouldn't be treated as a different repo.

::: hgext/overlay/tests/test-overlay-basic.t:104
(Diff revision 2)
> +  new file mode 100644
> +  --- /dev/null
> +  +++ b/foo
> +  @@ -0,0 +1,1 @@
> +  +foo
> +  

i'd like to see the test co tip and ensure the --into directory was created correctly.

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

https://reviewboard.mozilla.org/r/105688/#review109532

> do we care about python3 support here?
> iterkeys() won't work on py3.

Not yet since Mercurial itself doesn't work on Python 3.

Although in this case, the manifest class implements `iterkeys()` IIRC so it isn't an issue :)

> the protocol should probably be excluded from the path; switching protocols for the same repo shouldn't be treated as a different repo.

Good idea.

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

https://reviewboard.mozilla.org/r/105688/#review109532

> i'd like to see the test co tip and ensure the --into directory was created correctly.

This comment doesn't parse :/

Comment 7

[:glob ✱]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

https://reviewboard.mozilla.org/r/105688/#review109532

> This comment doesn't parse :/

sorry about that.  how about..

i'd like to see the test execute "hg co tip", and then ensure the directory specified by --into was created with the correct name and location.

Comment 8

[Gregory Szorc [:gps]]

Attachment: vcssync: command for performing hg repo overlaying (bug 1331697);

Now that we have a low-level `hg overlay` command, we need to put it
to use.

This commit introduces a Python API and CLI program for invoking
`hg overlay`. It lives at a layer above `hg overlay` and provides
functionality useful for the VCS syncing service. Although, the added
value seems somewhat minimal: I'm halfway tempted to roll the
functionality into `hg overlay`. The reason I didn't is because I
originally envisioned `hg overlay` being a low-level "plumbing"
command and not a core part of a version control "syncing" tool
per se.

Anyway, the added routine is basically a specialized version of
Autoland. You give it URLs to repos, the revision and sub-directory
in the destination for the overlay, and it does the rest. It can
even push the result when it is done.

`hg overlay` has no persistent state in the destination repository.
Instead, it uses changeset metadata to track what was processed.
This means that the overlay is mostly deterministic (only thing
that should change is parents the overlay is operated on). The
functionality in this commit relies on that mostly-deterministic
behavior. One of the first things done after invocation is to
`hg strip` mutable changesets. So if there were changesets from
a previous `hg overlay` sitting around, they are wiped away
and the operation is performed again. This conveniently means
that intermittent failures due to tree closures, push races, etc
just need to be retried and things should recover. Some of this
retry logic is built into the CLI interface. It isn't as robust
as Autoland. But I think that's OK for a first iteration.

Review commit: https://reviewboard.mozilla.org/r/110922/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/110922/

Comment 9

[Gregory Szorc [:gps]]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/105688/diff/2-3/

Comment 10

[Gregory Szorc [:gps]]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/105688/diff/3-4/

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8835234 [details]
vcssync: command for performing hg repo overlaying (bug 1331697);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/110922/diff/1-2/

Comment 12

[Gregory Szorc [:gps]]

glob: I'm probably not going to finish the Ansible work tonight, but the idea is to invoke the CLI process for doing the overlay after every push to https://hg.mozilla.org/projects/converted-servo-linear and periodically (on a systemd timer) as a fallback. The result will then get pushed somewhere, likely ssh://hg.mozilla.org/integration/autoland until we have somewhere better.

If trees are open, it should "just work" and commits to github.com/servo/servo will be linearized, converted to Mercurial, pushed to https://hg.mozilla.org/projects/converted-servo-linear then overlayed into the tip of a Firefox repo and pushed somewhere. The overlay operation should never fail if the code in this patch is working properly and we keep the servo/ directory of Firefox repos read-only outside of this automated process. The only thing that should fail is the push operation (due to e.g. tree closures). The current recourse for that is to wait for the timer to retry the overlay. I think that's good enough for a MVP.

The code is definitely reusable outside the context of Servo (this was intentional). It should also be possible to fold the new code in mozvcssync/overlay.py into Autoland and/or the eventual Servo landing tool with minimal effort.

I'll try to linger on IRC tonight in case you want to chat about things.

Comment 13

[:glob ✱]

Comment on attachment 8828223 [details]
overlay: command for overlaying a repo into a subdirectory of another (bug 1331697);

https://reviewboard.mozilla.org/r/105688/#review112260

Comment 14

[:glob ✱]

Comment on attachment 8835234 [details]
vcssync: command for performing hg repo overlaying (bug 1331697);

https://reviewboard.mozilla.org/r/110922/#review112518

Comment 15

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0ff7e64be514
overlay: command for overlaying a repo into a subdirectory of another ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a355883e91b7
vcssync: command for performing hg repo overlaying ; r=glob