1391394 - Add sparse checkout support to robustcheckout

Status: RESOLVED FIXED

(Developer Services :: Mercurial: robustcheckout, enhancement)

Description

[Gregory Szorc [:gps]]

Mercurial 4.3 supports sparse checkouts (working directories only containing a subset of files). Use of sparse checkouts will allow us to not have to realize a few hundred thousand files for many CI tasks. This will effectively enable us to use checkouts in tasks where that was performance prohibitive before.

Comment 1

[Gregory Szorc [:gps]]

Attachment: robustcheckout: update command help;

Review commit: https://reviewboard.mozilla.org/r/171412/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/171412/

Comment 2

[Gregory Szorc [:gps]]

Attachment: robustcheckout: support sparse profiles (bug 1391394);

Mercurial 4.3 has an experimental sparse checkout feature. It allows
the working directory to consist of a subset of files from a
revision.

There are a number of different ways to specify what to include in
the sparse checkout. But the method we care about supporting for
Firefox is "profiles." Essentially, an in-tree file can define
patterns of files to include or exclude in the working directory.
You "activate" a profile and Mercurial reads that file and applies
the patterns to the working directory.

In this commit, we teach robustcheckout to perform sparse checkouts
using sparse profile files. While Mercurial supports multiple
active profiles, we make the arbitrary decision to only allow a
single profile. And, when we change profiles, we deactivate the old
one and enable the new one rather than union them. This avoids
all kinds of non-determinism and potential wonkiness around
combinations of profiles that don't play well together. This should
therefore eliminate a whole class of bugs.

Another area where we've made arbitrary decisions is around enabling
and disabling sparse on existing checkouts. This isn't strictly
needed. But I want robustcheckout to be very conservative in its
sparse support initially. I don't want CI mixing sparse with non
sparse. And this restriction forces that.

The code in robustcheckout for supporting sparse is somewhat hacky
and some things are sub-optimal. For example, when changing profiles
it does a working directory update to the new profile and then
potentially a separate working directory update to the new revision.
Unfortunately, Mercurial's internal APIs don't allow you to write
out the new profile without the working directory update. We /could/
copy some code to facilitate this. But my inclination is to go with
the simple (albeit sub-optimal) solution and then add complexity
only if needed. Another example is having to monkeypatch the
dirstate object so purge purges files not in the sparse config.

Because of the level of the hacks and the experimental nature of
sparse in Mercurial, the feature will only run on Mercurial 4.3.
When Mercurial 4.4 is out, we'll have to explicitly tell
robustcheckout to support it.

Review commit: https://reviewboard.mozilla.org/r/171414/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/171414/

Comment 3

[:glob ✱]

Comment on attachment 8900070 [details]
robustcheckout: support sparse profiles (bug 1391394);

https://reviewboard.mozilla.org/r/171414/#review176606

::: hgext/robustcheckout/__init__.py:546
(Diff revision 1)
>  
> +        # Mercurial 4.3 doesn't purge files outside the sparse checkout.
> +        # See https://bz.mercurial-scm.org/show_bug.cgi?id=5626. Force
> +        # purging by monkeypatching the sparse matcher.
> +        try:
> +            oldsparsefn = getattr(repo.dirstate, '_sparsematchfn', None)

nit: old_sparse_fn is much easier to read.

Comment 4

[:glob ✱]

Comment on attachment 8900069 [details]
robustcheckout: update command help;

https://reviewboard.mozilla.org/r/171412/#review176622

Comment 5

[Gregory Szorc [:gps]]

Comment on attachment 8900070 [details]
robustcheckout: support sparse profiles (bug 1391394);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/171414/diff/1-2/

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e5c5809605e7
robustcheckout: support sparse profiles ; r=glob

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0451f128550d
robustcheckout: tweak log message to reflect reality