1333616 - Deploy Mercurial 4.1 to hg.mozilla.org

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

Mercurial 4.1 will be released ~February 1. Per our release policy, we're eligible to roll it out after the first point release, which should be ~March 1.

Since I'll be away until the middle of March, we likely won't deploy 4.1 until ~March 20.

Mercurial 4.1 contains support for zstd compression over the wire protocol (as opposed to zlib). My measurements show this can reduce server-side CPU usage to ~60% compared to zlib. This should make a noticeable difference for serving speed and capacity. Of course, we need Mercurial 4.1 clients deployed to take advantage of that.

Comment 1

[Gregory Szorc [:gps]]

I'm back. This is overdue. I want to get it rolled out.

Comment 2

[Gregory Szorc [:gps]]

Attachment: testing: implement hghave checks for Mercurial 4.0+ and 4.1+ (bug 1333616);

This is in preparation of upgrading to 4.1.

Review commit: https://reviewboard.mozilla.org/r/125670/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125670/

Comment 3

[Gregory Szorc [:gps]]

Attachment: vcssync: require Mercurial 4.0+ to run hg related tests (bug 1333616);

We wrote this code with 4.0 in mind. The tests mostly pass on 3.8
and 3.9. But I don't want to give the illusion of support. So let's
explicitly require Mercurial 4.0+ to run all Mercurial related
vcssync tests.

Review commit: https://reviewboard.mozilla.org/r/125672/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125672/

Comment 4

[Gregory Szorc [:gps]]

Attachment: testing: don't install Mercurial 3.7 (bug 1333616);

Our policy is to only support the last 4 releases. We're about to
support 4.1, so stop installing 3.7.

Review commit: https://reviewboard.mozilla.org/r/125674/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125674/

Comment 5

[Gregory Szorc [:gps]]

Attachment: hgext: drop support for Mercurial 3.7 (bug 1333616);

Our policy is to only support the last 4 Mercurial releases. 4.1
is out, which means we now require 3.8. So drop explicit compatibility
with 3.7. In some cases, extensions will now fail to load on 3.7 due
to a bumpbed "minimumhgversion." This is by design: it allows us to
make changes without having to be compatible with 3.7.

Review commit: https://reviewboard.mozilla.org/r/125676/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125676/

Comment 6

[Gregory Szorc [:gps]]

Attachment: hghooks: make changelog_correctness work with 4.1 (bug 1333616);

The API for manifest access completely changed in Mercurial 4.1. Make
the hook compatible with old and new APIs.

I /think/ the new API has methods that can perform a lot of what this
hook is doing manually. We should look into porting the hook to
use the built-in API. An inline TODO has been added so this fact
doesn't get lost.

Review commit: https://reviewboard.mozilla.org/r/125678/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125678/

Comment 7

[Gregory Szorc [:gps]]

Attachment: vcssync: temporarily do not set convert.git.saverev (bug 1333616);

This config option was introduced in Mercurial 4.1 and has no effect on
Mercurial 4.0. In order to demonstrate that the Mercurial 4.1 upgrade
doesn't change tests without this option in play, this commit
disables the config option so we can perform the Mercurial upgrade
without extra test changes.

Review commit: https://reviewboard.mozilla.org/r/125680/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125680/

Comment 8

[Gregory Szorc [:gps]]

Attachment: testing: install Mercurial 4.1.2 in local environment (bug 1333616);

This is pretty straightforward. We only bump the test environment
for now. Upgrading servers will come later.

Review commit: https://reviewboard.mozilla.org/r/125682/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125682/

Comment 9

[Gregory Szorc [:gps]]

Attachment: vcssync: bump requirement to Mercurial 4.1 (bug 1333616);

We currently run vcssync in production with Mercurial 4.1.
Previously, the tests only ran against Mercurial 4.0 and had
expectations tied to Mercurial 4.0. This notably included the
expectation that "convert.git.saverev=false" is a no-op
on 4.0.

Since the upgrade to 4.1 in the test environment is complete,
let's bump the test requirements from 4.0 to 4.1. I also snuck
in a bump to the overlay extension's minimum version requirement.

With "convert.git.saverev=false" being temporarily commented
out, there are no test changes between 4.0 and 4.1, which of course
is good.

Review commit: https://reviewboard.mozilla.org/r/125684/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125684/

Comment 10

[Gregory Szorc [:gps]]

Attachment: vcssync: re-enable convert.git.saverev=false (bug 1333616);

We successfully performed the 4.0 to 4.1 upgrade in the test
environment without any test fallout to vcssync. But this was
because we disabled convert.git.saverev=false.

In this commit, we re-enable that config option (which only works
on 4.1). As test-linearize-git-to-hg-source-annotations.t shows,
the "convert_revision" extra metadata disappeared (as the inline TODO
said it would). Hashes in other tests also changed because the
removal of this metadata changed the changeset content which changed
the hash. All other functionality remains unchanged.

Review commit: https://reviewboard.mozilla.org/r/125686/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/125686/

Comment 11

[Gregory Szorc [:gps]]

Attachment: ansible/autoland: use mercurial-yum role to install Mercurial (bug 1333616);

The code for installing Mercurial in the autoland role is redundant
with code in the mercurial-yum role. Let's consolidate it.

Review commit: https://reviewboard.mozilla.org/r/126166/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126166/

Comment 12

[Gregory Szorc [:gps]]

Attachment: serverlog: only mark as compatible against 4.0;

We don't care about running this extension outside of servers. Let's
mark it as only compatible against what we run on servers.

Review commit: https://reviewboard.mozilla.org/r/126168/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126168/

Comment 13

[Gregory Szorc [:gps]]

Attachment: hgserver: glob over bundle type in clonebundles test (bug 1333616);

Mercurial 4.1 will prefer a zstd bundle over a gzip bundle. Rather than
make the test conditional on version, let's just glob out the bundle
type.

Review commit: https://reviewboard.mozilla.org/r/126170/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126170/

Comment 14

[Gregory Szorc [:gps]]

Attachment: ansible/vcs-sync: upgrade Mercurial to 4.1.2 (bug 1333616);

Let's get the latest version of Mercurial 4.1 installed so we can
standardize on that version.

Review commit: https://reviewboard.mozilla.org/r/126172/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126172/

Comment 15

[Gregory Szorc [:gps]]

Attachment: vcssync: require Mercurial 4.1 (bug 1333616);

The code for vcssync actually rely on a feature in 4.1. But setup.py
allows 4.0 because that's what we were running in our test environment.
Now that we've upgraded the test environment and made the tests pass
on 4.1, let's bump the install requirement to 4.1+.

Review commit: https://reviewboard.mozilla.org/r/126174/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126174/

Comment 16

[Gregory Szorc [:gps]]

Attachment: ansible: mass upgrade to Mercurial 4.1.2 (bug 1333616);

All virtualenvs and system package installs have been bumped to
Mercurial 4.1.2.

Review commit: https://reviewboard.mozilla.org/r/126176/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126176/

Comment 17

[Gregory Szorc [:gps]]

Attachment: hgserver: update test-obsolescence.t to be compatible with 4.1 (bug 1333616);

This test's output changed with Mercurial 4.1 because the HTTP server
now exposes additional capabilities advertising protocol and
compression support. This allows clients to perform a passive
negotiation and choose the most optimal protocol version and
compression that both sides support.

Review commit: https://reviewboard.mozilla.org/r/126178/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126178/

Comment 18

[Gregory Szorc [:gps]]

Attachment: hgext: mark most extensions as compatible with 4.1 (bug 1333616);

Review commit: https://reviewboard.mozilla.org/r/126180/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126180/

Comment 19

[Gregory Szorc [:gps]]

Attachment: hgext: mark server-side extensions as only compatible with 4.1 (bug 1333616);

Now that we've upgraded all servers to Mercurial 4.1.2, we mark
server-side extensions as only compatible with this version. We just
don't care strongly about running these extensions outside our server
environment.

Review commit: https://reviewboard.mozilla.org/r/126182/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126182/

Comment 20

[Gregory Szorc [:gps]]

Attachment: hgtemplates: synchronize templates with Mercurial 4.1.2 (bug 1333616);

We synchronize Mercurial's built-in templates with 4.1.2. A subsequent
commit will apply gitweb changes to the gitweb_mozilla style.

Review commit: https://reviewboard.mozilla.org/r/126184/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126184/

Comment 21

[Gregory Szorc [:gps]]

Attachment: hgtemplates: port 4.1 template changes to gitweb_mozilla (bug 1333616);

gitweb_mozilla is our fork of the gitweb style. Every release, we have
to port changes to the gitweb style to gitweb_mozilla.

The changes in this release weren't too bad.

process_dates() was factored into mercurial.js, so we can delete it.

In order to support CSP, all <script> instances now have some template
boilerplate to automatically emit a CSP nonce.

The only non-obvious change was pushlog.tmpl had a <script> that needed
to be taught about a nonce.

Review commit: https://reviewboard.mozilla.org/r/126186/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126186/

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8853600 [details]
testing: implement hghave checks for Mercurial 4.0+ and 4.1+ (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125670/diff/1-2/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8853601 [details]
vcssync: require Mercurial 4.0+ to run hg related tests (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125672/diff/1-2/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8853602 [details]
testing: don't install Mercurial 3.7 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125674/diff/1-2/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8853603 [details]
hgext: drop support for Mercurial 3.7 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125676/diff/1-2/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8853604 [details]
hghooks: make changelog_correctness work with 4.1 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125678/diff/1-2/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8853605 [details]
vcssync: temporarily do not set convert.git.saverev (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125680/diff/1-2/

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8853606 [details]
testing: install Mercurial 4.1.2 in local environment (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125682/diff/1-2/

Comment 29

[Gregory Szorc [:gps]]

Comment on attachment 8853607 [details]
vcssync: bump requirement to Mercurial 4.1 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125684/diff/1-2/

Comment 30

[Gregory Szorc [:gps]]

Comment on attachment 8853608 [details]
vcssync: re-enable convert.git.saverev=false (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/125686/diff/1-2/

Comment 31

[Gregory Szorc [:gps]]

Attachment: hgserver: update test-http-post-args.t for Mercurial 4.1 (bug 1333616);

The total size of the HTTP message changed because 4.1 added some more
request headers. Update the test accordingly.

Review commit: https://reviewboard.mozilla.org/r/126188/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/126188/

Comment 32

[Gregory Szorc [:gps]]

Comment on attachment 8854224 [details]
hgext: mark most extensions as compatible with 4.1 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/126180/diff/1-2/

Comment 33

[Gregory Szorc [:gps]]

Comment on attachment 8854225 [details]
hgext: mark server-side extensions as only compatible with 4.1 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/126182/diff/1-2/

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 8854226 [details]
hgtemplates: synchronize templates with Mercurial 4.1.2 (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/126184/diff/1-2/

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 8854227 [details]
hgtemplates: port 4.1 template changes to gitweb_mozilla (bug 1333616);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/126186/diff/1-2/

Comment 36

[:glob ✱]

Comment on attachment 8854218 [details]
serverlog: only mark as compatible against 4.0;

https://reviewboard.mozilla.org/r/126168/#review128954

Comment 37

[:glob ✱]

Comment on attachment 8853600 [details]
testing: implement hghave checks for Mercurial 4.0+ and 4.1+ (bug 1333616);

https://reviewboard.mozilla.org/r/125670/#review128958

Comment 38

[:glob ✱]

Comment on attachment 8854217 [details]
ansible/autoland: use mercurial-yum role to install Mercurial (bug 1333616);

https://reviewboard.mozilla.org/r/126166/#review128968

Comment 39

[:glob ✱]

Comment on attachment 8853601 [details]
vcssync: require Mercurial 4.0+ to run hg related tests (bug 1333616);

https://reviewboard.mozilla.org/r/125672/#review128972

Comment 40

[:glob ✱]

Comment on attachment 8853602 [details]
testing: don't install Mercurial 3.7 (bug 1333616);

https://reviewboard.mozilla.org/r/125674/#review129316

Comment 41

[:glob ✱]

Comment on attachment 8853603 [details]
hgext: drop support for Mercurial 3.7 (bug 1333616);

https://reviewboard.mozilla.org/r/125676/#review129318

Comment 42

[:glob ✱]

Comment on attachment 8853604 [details]
hghooks: make changelog_correctness work with 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/125678/#review129320

Comment 43

[:glob ✱]

Comment on attachment 8854219 [details]
hgserver: glob over bundle type in clonebundles test (bug 1333616);

https://reviewboard.mozilla.org/r/126170/#review129322

Comment 44

[:glob ✱]

Comment on attachment 8853605 [details]
vcssync: temporarily do not set convert.git.saverev (bug 1333616);

https://reviewboard.mozilla.org/r/125680/#review129324

Comment 45

[:glob ✱]

Comment on attachment 8853606 [details]
testing: install Mercurial 4.1.2 in local environment (bug 1333616);

https://reviewboard.mozilla.org/r/125682/#review129326

Comment 46

[:glob ✱]

Comment on attachment 8853607 [details]
vcssync: bump requirement to Mercurial 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/125684/#review129448

Comment 47

[:glob ✱]

Comment on attachment 8853608 [details]
vcssync: re-enable convert.git.saverev=false (bug 1333616);

https://reviewboard.mozilla.org/r/125686/#review129454

Comment 48

[:glob ✱]

Comment on attachment 8854220 [details]
ansible/vcs-sync: upgrade Mercurial to 4.1.2 (bug 1333616);

https://reviewboard.mozilla.org/r/126172/#review129456

Comment 49

[:glob ✱]

Comment on attachment 8854221 [details]
vcssync: require Mercurial 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/126174/#review129458

Comment 50

[:glob ✱]

Comment on attachment 8854222 [details]
ansible: mass upgrade to Mercurial 4.1.2 (bug 1333616);

https://reviewboard.mozilla.org/r/126176/#review129462

Comment 51

[:glob ✱]

Comment on attachment 8854223 [details]
hgserver: update test-obsolescence.t to be compatible with 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/126178/#review129464

Comment 52

[:glob ✱]

Comment on attachment 8854230 [details]
hgserver: update test-http-post-args.t for Mercurial 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/126188/#review129466

Comment 53

[:glob ✱]

Comment on attachment 8854224 [details]
hgext: mark most extensions as compatible with 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/126180/#review129468

Comment 54

[:glob ✱]

Comment on attachment 8854225 [details]
hgext: mark server-side extensions as only compatible with 4.1 (bug 1333616);

https://reviewboard.mozilla.org/r/126182/#review129478

Comment 55

[:glob ✱]

Comment on attachment 8854226 [details]
hgtemplates: synchronize templates with Mercurial 4.1.2 (bug 1333616);

https://reviewboard.mozilla.org/r/126184/#review129484

Comment 56

[:glob ✱]

Comment on attachment 8854227 [details]
hgtemplates: port 4.1 template changes to gitweb_mozilla (bug 1333616);

https://reviewboard.mozilla.org/r/126186/#review129486

Comment 57

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8d24832ce38b
ansible/autoland: use mercurial-yum role to install Mercurial ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/78cc21d65f62
testing: implement hghave checks for Mercurial 4.0+ and 4.1+ ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9e969588d0b7
vcssync: require Mercurial 4.0+ to run hg related tests ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/23de0ccc7f45
testing: don't install Mercurial 3.7 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a638dc739d1d
hgext: drop support for Mercurial 3.7 ; r=glob

Comment 58

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/8c237cebadb2
hghooks: make changelog_correctness work with 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a9f5f32e8990
hgserver: glob over bundle type in clonebundles test ; r=glob

Comment 59

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/314ce18c7da7
vcssync: temporarily do not set convert.git.saverev ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a7c33a8decf5
testing: install Mercurial 4.1.2 in local environment ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/917c02d0b3c2
vcssync: bump requirement to Mercurial 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/aabea2938055
vcssync: re-enable convert.git.saverev=false ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6d7e0ccb2689
ansible/vcs-sync: upgrade Mercurial to 4.1.2 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f0573e726890
vcssync: require Mercurial 4.1 ; r=glob

Comment 60

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0145a61ce6c3
ansible: mass upgrade to Mercurial 4.1.2 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/717707a86380
hgserver: update test-obsolescence.t to be compatible with 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/619c4455880b
hgserver: update test-http-post-args.t for Mercurial 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/c839fcfefb21
hgext: mark most extensions as compatible with 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e54c599f4ed1
hgext: mark server-side extensions as only compatible with 4.1 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/90af27d1954e
hgtemplates: synchronize templates with Mercurial 4.1.2 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/68c770b15f56
hgtemplates: port 4.1 template changes to gitweb_mozilla ; r=glob

Comment 61

[Gregory Szorc [:gps]]

It will likely be another 2-4 hours before I deploy this since I'll be bouncing around meetings and want to be anchored in front of my computer when this goes out.

Comment 62

[Gregory Szorc [:gps]]

4.1.2 is completely deployed to hg.mozilla.org.

Comment 63

[Mozilla Autoland]

We're sorry, Autoland could not rebase your commits for you automatically. Please manually rebase your commits and try again.

hg error in cmd: hg rebase -s 378d1cf7e340 -d b895763b7fe7: rebasing 5233:378d1cf7e340 "ansible/autoland: use mercurial-yum role to install Mercurial (bug 1333616); r=glob"
note: rebase of 5233:378d1cf7e340 created no changes to commit
rebasing 5234:be5f03b28fd3 "serverlog: only mark as compatible against 4.0; r=glob"
merging hgext/serverlog/__init__.py
warning: conflicts while merging hgext/serverlog/__init__.py! (edit, then use 'hg resolve --mark')
unresolved conflicts (see hg resolve, then hg rebase --continue)