1228088 - Upgrade hg.mozilla.org to Mercurial 3.7

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

Per our deployment schedule, this should happen around March 1, 2016.

Filing the bug now to track backwards compatibility concerns.

3.7 creates generaldelta repos by default. We likely want to create non-generaldelta repos on the server. Otherwise legacy clients requesting data from generaldelta repos will blow up CPU on the server. I doubt we'll be able to mass convert repos to generaldelta until end of 2016 at the earliest.

Comment 1

[Gregory Szorc [:gps]]

We'll want to globally set format.usegeneraldelta=false to prevent new repos from being created with generaldelta. At some point (once most clients are using bundle2), we'll want to consider mass converting repos to generaldelta to get space and CPU savings.

Comment 2

[Gregory Szorc [:gps]]

The upgrade is tentatively scheduled for March 7.

Comment 3

[Gregory Szorc [:gps]]

Attachment: MozReview Request: mozext: use bookmarks.recordchange (bug 1228088); r=dminor

Mercurial 3.7 throws a warning when bookmarks.write() is called.
Call bookmarks.recordchange() instead. This API has been around since at
least Mercurial 3.3.

Review commit: https://reviewboard.mozilla.org/r/37297/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37297/

Comment 4

[Gregory Szorc [:gps]]

Attachment: MozReview Request: mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

Mercurial 3.7 prints a warning when bookmarks.write() is called.
Switch to bookmarks.recordchange(), which is the preferred API.

Review commit: https://reviewboard.mozilla.org/r/37299/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37299/

Comment 5

[Gregory Szorc [:gps]]

Attachment: MozReview Request: testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

Let's upgrade the test environment to 3.7 before we switch over the
Ansible infrastructure.

Review commit: https://reviewboard.mozilla.org/r/37301/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37301/

Comment 6

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

Mercurial 3.7 creates generaldelta repositories by default. This can be
problematic on hg.mozilla.org.

Mercurial clients that don't support bundle2 require servers serving
generaldelta repositories to re-encode the data to non-generaldelta.
This can require a lot of CPU, especially for large repositories.

Eventually, we'll want to use generaldelta on hg.mozilla.org. However,
until most clients are modern, we risk a CPU explosion when serving
generaldelta repos to legacy clients. To play it safe, we disable
generaldelta on new repositories by default.

Review commit: https://reviewboard.mozilla.org/r/37303/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37303/

Comment 7

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

Mercurial 3.7 removed the clone bundles server-side advertisement.
So let's stop testing that it appears.

Review commit: https://reviewboard.mozilla.org/r/37305/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37305/

Comment 8

[Gregory Szorc [:gps]]

Attachment: MozReview Request: ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r?fubar

TODO use 3.7.2 when it is released.

We mass upgrade the world (except for Autoland) to Mercurial 3.7.

Review commit: https://reviewboard.mozilla.org/r/37307/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37307/

Comment 9

[Gregory Szorc [:gps]]

Attachment: MozReview Request: reviewboard: prune output not relevant to test; r=dminor

`hg histedit` has become less chatty in modern versions of Mercurial.
This was causing tests to fail due to inconsistent output between
versions.

This commit greps out output that varies between versions so tests pass.

Review commit: https://reviewboard.mozilla.org/r/37309/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37309/

Comment 10

[Gregory Szorc [:gps]]

Attachment: MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

Mercurial 3.7 is issuing warnings when running tests that the revset
functions in mozext are returning lists instead of revset classes.

mozext was one of the first Mercurial extensions I wrote. It is one of
the oldest and doesn't exactly contain the best patterns or modern API
usage.

This commit brings the revset code into the modern age. Most of the
changes involve calling subset.filter() to lazily evaluate a revset.
Before, we iterated over the subset, which meant that we had to evaluate
every element before returning. This almost certainly made a number of
revset queries slower than they should have been.

Review commit: https://reviewboard.mozilla.org/r/37311/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37311/

Comment 11

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

parent changed from a generator to a lambda. A number of other keys were
added. We filter them to keep test output the same. We should consider
displaying them some day...

Review commit: https://reviewboard.mozilla.org/r/37313/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37313/

Comment 12

[Gregory Szorc [:gps]]

Attachment: MozReview Request: bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

The bundleclone extension no longer works in Mercurial 3.7 because
localrepository.clone (the function it overrides to do most of the
special clone work) no longer exists.

The extension no-ops when the server supports clone bundles anyway. So
it was already doing nothing on 3.6 when clone bundles was enabled.

We update the tests to not run on 3.7+. We also add a new test that
verifies 3.7+ continues to work (as a no-op).

Review commit: https://reviewboard.mozilla.org/r/37315/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37315/

Comment 13

[Gregory Szorc [:gps]]

Attachment: MozReview Request: pushrebase: upgrade extension

Imported changeset f9f888fb96f4 from the hg-experimental upstream repo.
The test has been modified to be compatible with the new version.

The "testedwith" value in the extension has been updated to reflect
reality (the extension only works with Mercurial 3.7 it appears).

Review commit: https://reviewboard.mozilla.org/r/37317/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37317/

Comment 14

[Kendall Libby [:fubar] (he/him)]

Comment on attachment 8725061 [details]
MozReview Request: ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r?fubar

https://reviewboard.mozilla.org/r/37307/#review33923

lgtm

Comment 15

[Dan Minor [:dminor]]

Comment on attachment 8725056 [details]
MozReview Request: mozext: use bookmarks.recordchange (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37297/#review33927

Comment 16

[Dan Minor [:dminor]]

Comment on attachment 8725057 [details]
MozReview Request: mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37299/#review33931

Comment 17

[Dan Minor [:dminor]]

Comment on attachment 8725058 [details]
MozReview Request: testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37301/#review33933

Comment 18

[Dan Minor [:dminor]]

Comment on attachment 8725059 [details]
MozReview Request: ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37303/#review33935

Comment 19

[Dan Minor [:dminor]]

Comment on attachment 8725060 [details]
MozReview Request: hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37305/#review33937

Comment 20

[Dan Minor [:dminor]]

Comment on attachment 8725062 [details]
MozReview Request: reviewboard: prune output not relevant to test; r=dminor

https://reviewboard.mozilla.org/r/37309/#review33939

Comment 21

[Dan Minor [:dminor]]

Comment on attachment 8725063 [details]
MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37311/#review33947

::: hgext/mozext/__init__.py:882
(Diff revision 1)
> +    def filter(x):

There's a builtin function called filter(), please use a different name.

Comment 22

[Dan Minor [:dminor]]

Comment on attachment 8725064 [details]
MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37313/#review33951

::: hgext/hgmo/__init__.py:391
(Diff revision 1)
> -            elif k in ('bookmarks', 'branches', 'changelogtag', 'child', 'inbranch', 'tags'):
> +            elif k in ('bookmarks', 'branch', 'branches', 'changelogtag', 'child', 'inbranch', 'phase', 'tags'):

Please move the deletes to the top of the control structure, I think it will improve readability as you have to add more special cases.

Comment 23

[Dan Minor [:dminor]]

Comment on attachment 8725065 [details]
MozReview Request: bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37315/#review33953

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8725056 [details]
MozReview Request: mozext: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37297/diff/1-2/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8725057 [details]
MozReview Request: mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37299/diff/1-2/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8725058 [details]
MozReview Request: testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37301/diff/1-2/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8725059 [details]
MozReview Request: ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37303/diff/1-2/

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8725060 [details]
MozReview Request: hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37305/diff/1-2/

Comment 29

[Gregory Szorc [:gps]]

Comment on attachment 8725061 [details]
MozReview Request: ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r?fubar

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37307/diff/1-2/

Comment 30

[Gregory Szorc [:gps]]

Comment on attachment 8725062 [details]
MozReview Request: reviewboard: prune output not relevant to test; r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37309/diff/1-2/

Comment 31

[Gregory Szorc [:gps]]

Comment on attachment 8725063 [details]
MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37311/diff/1-2/

Comment 32

[Gregory Szorc [:gps]]

Comment on attachment 8725064 [details]
MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37313/diff/1-2/

Comment 33

[Gregory Szorc [:gps]]

Comment on attachment 8725065 [details]
MozReview Request: bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37315/diff/1-2/

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 8725066 [details]
MozReview Request: pushrebase: upgrade extension

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37317/diff/1-2/

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 8725056 [details]
MozReview Request: mozext: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37297/diff/2-3/

Comment 36

[Gregory Szorc [:gps]]

Comment on attachment 8725057 [details]
MozReview Request: mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37299/diff/2-3/

Comment 37

[Gregory Szorc [:gps]]

Comment on attachment 8725058 [details]
MozReview Request: testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37301/diff/2-3/

Comment 38

[Gregory Szorc [:gps]]

Comment on attachment 8725059 [details]
MozReview Request: ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37303/diff/2-3/

Comment 39

[Gregory Szorc [:gps]]

Comment on attachment 8725060 [details]
MozReview Request: hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37305/diff/2-3/

Comment 40

[Gregory Szorc [:gps]]

Comment on attachment 8725061 [details]
MozReview Request: ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r?fubar

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37307/diff/2-3/

Comment 41

[Gregory Szorc [:gps]]

Comment on attachment 8725062 [details]
MozReview Request: reviewboard: prune output not relevant to test; r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37309/diff/2-3/

Comment 42

[Gregory Szorc [:gps]]

Comment on attachment 8725063 [details]
MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37311/diff/2-3/

Comment 43

[Gregory Szorc [:gps]]

Comment on attachment 8725064 [details]
MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37313/diff/2-3/

Comment 44

[Gregory Szorc [:gps]]

Comment on attachment 8725065 [details]
MozReview Request: bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37315/diff/2-3/

Comment 45

[Gregory Szorc [:gps]]

Comment on attachment 8725066 [details]
MozReview Request: pushrebase: upgrade extension

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37317/diff/2-3/

Comment 46

[Dan Minor [:dminor]]

Comment on attachment 8725063 [details]
MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37311/#review34111

Comment 47

[Dan Minor [:dminor]]

Comment on attachment 8725064 [details]
MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

https://reviewboard.mozilla.org/r/37313/#review34113

Comment 48

[Gregory Szorc [:gps]]

Comment on attachment 8725056 [details]
MozReview Request: mozext: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37297/diff/3-4/

Comment 49

[Gregory Szorc [:gps]]

Comment on attachment 8725057 [details]
MozReview Request: mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37299/diff/3-4/

Comment 50

[Gregory Szorc [:gps]]

Comment on attachment 8725058 [details]
MozReview Request: testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37301/diff/3-4/

Comment 51

[Gregory Szorc [:gps]]

Comment on attachment 8725059 [details]
MozReview Request: ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37303/diff/3-4/

Comment 52

[Gregory Szorc [:gps]]

Comment on attachment 8725060 [details]
MozReview Request: hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37305/diff/3-4/

Comment 53

[Gregory Szorc [:gps]]

Comment on attachment 8725061 [details]
MozReview Request: ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r?fubar

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37307/diff/3-4/

Comment 54

[Gregory Szorc [:gps]]

Comment on attachment 8725062 [details]
MozReview Request: reviewboard: prune output not relevant to test; r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37309/diff/3-4/

Comment 55

[Gregory Szorc [:gps]]

Comment on attachment 8725063 [details]
MozReview Request: mozext: use modern revset APIs (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37311/diff/3-4/

Comment 56

[Gregory Szorc [:gps]]

Comment on attachment 8725064 [details]
MozReview Request: hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37313/diff/3-4/

Comment 57

[Gregory Szorc [:gps]]

Comment on attachment 8725065 [details]
MozReview Request: bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37315/diff/3-4/

Comment 58

[Gregory Szorc [:gps]]

Comment on attachment 8725066 [details]
MozReview Request: pushrebase: upgrade extension

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/37317/diff/3-4/

Comment 59

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgtemplates: synchronize templates from Mercurial 3.7.2 (bug 1228088); r?dminor

Templates from the Mercurial 3.7.2 release have been applied.

This should be a rubber stamp review, since we don't modify any of these
files.

Review commit: https://reviewboard.mozilla.org/r/37681/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37681/

Comment 60

[Gregory Szorc [:gps]]

Attachment: MozReview Request: hgtemplates: incorporate gitweb theme changes into gitweb_mozilla (bug 1228088); r?dminor

Mostly RSS and ATOM foo. Nothing too serious by the looks of it.

Review commit: https://reviewboard.mozilla.org/r/37683/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/37683/

Comment 61

[Dan Minor [:dminor]]

Comment on attachment 8725890 [details]
MozReview Request: hgtemplates: synchronize templates from Mercurial 3.7.2 (bug 1228088); r?dminor

https://reviewboard.mozilla.org/r/37681/#review34387

Comment 62

[Dan Minor [:dminor]]

Comment on attachment 8725891 [details]
MozReview Request: hgtemplates: incorporate gitweb theme changes into gitweb_mozilla (bug 1228088); r?dminor

https://reviewboard.mozilla.org/r/37683/#review34389

Comment 63

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/426d0e678374586ae615c6fafcf96706b5c1e3a1
mozext: use bookmarks.recordchange (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/a091ebaf6accb3601b3856da4ccf3bdd80b5eedc
mozhg: use bookmarks.recordchange (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/80570c2b3a9dae0a9b5c84dfb42ed024bd461deb
testing: use Mercurial 3.7.2 in test environment (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/edc7cb44cd7c724bfc4810a9c52fd406c0e6ee3c
ansible/hg: disable generaldelta on new repos (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/be2895740fbdf17f18244da05efd8e1e18f8973a
hgserver: remove test for clone bundles advertisement (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/559836f39d9a8c74aed702d7aa2c13775558d5b2
ansible: upgrade to Mercurial 3.7.2 (bug 1228088); r=fubar

https://hg.mozilla.org/hgcustom/version-control-tools/rev/1d0b011a119f3c57fb7192f33300b0be5473d346
mozext: use modern revset APIs (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/3f78fde149c210a797d2541fc9b5f2768009b6a3
hgmo: make automationrelevance API compatible with Mercurial 3.7 (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/aac2644370fde517e857eaa79cd2f4e10932a59c
bundleclone: make testing compatible with Mercurial 3.7 (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/24a1062cc93832448cc7d39623d394d5e532f796
hgtemplates: synchronize templates from Mercurial 3.7.2 (bug 1228088); r=dminor

https://hg.mozilla.org/hgcustom/version-control-tools/rev/6d36f5109c50a3920a3656c25709f85ef081fade
hgtemplates: incorporate gitweb theme changes into gitweb_mozilla (bug 1228088); r=dminor

Comment 64

[Gregory Szorc [:gps]]

hg.mozilla.org is now running Mercurial 3.7.2. Upgrade appeared to go off without any major hitches.