1423741 - Upgrade hg.mozilla.org to Mercurial 4.3

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

We're still running Mercurial 4.2 on hg.mozilla.org. Mercurial 4.4.2 is out. Let's get hg.mozilla.org on Mercurial 4.3 as a milestone towards 4.4.

Comment 1

[Gregory Szorc [:gps]]

Attachment: global: mass upgrade to PyYAML 3.12;

Let's use the latest version everywhere.

Review commit: https://reviewboard.mozilla.org/r/206050/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206050/

Comment 2

[Gregory Szorc [:gps]]

Attachment: hg-web: use latest version of certifi package;

Keeping CA certificates up to date seems like a good idea.

I don't think we actually do many (any?) outbound HTTP requests.
But it shouldn't hurt to have the latest certificates.

Review commit: https://reviewboard.mozilla.org/r/206052/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206052/

Comment 3

[Gregory Szorc [:gps]]

Attachment: hg-web: upgrade mod-wsgi package to latest;

Looks like there are a few bug fixes upstream. No major changes
reported in the release notes.

Review commit: https://reviewboard.mozilla.org/r/206054/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206054/

Comment 4

[Gregory Szorc [:gps]]

Attachment: hg-web: upgrade Mercurial in vcsreplicator environment to 4.3.3 (bug 1423741);

All vcsreplicator tests pass after the upgrade. Yay!

Review commit: https://reviewboard.mozilla.org/r/206056/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206056/

Comment 5

[Gregory Szorc [:gps]]

Attachment: hg-web: upgrade to Mercurial 4.3.3 in tools venv (bug 1423741);

Not many things rely on this Mercurial instance. No tests fail as a
result of this upgrade. So it seems safe.

Review commit: https://reviewboard.mozilla.org/r/206058/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206058/

Comment 6

[Gregory Szorc [:gps]]

Attachment: hgmo: add test for behavior when pulling empty manifest;

We have some 0 byte clonebundles.manifest files sitting around in
production. The upgrade to 4.3 found an issue with this code.
So while I'm here, let's clean things up a little.

Review commit: https://reviewboard.mozilla.org/r/206060/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206060/

Comment 7

[Gregory Szorc [:gps]]

Attachment: hgmo: remove clonebundles.manifest when not present on server;

There are a few repos on hg.mozilla.org that have 0 byte
clonebundles.manifest files. I think it is better behavior to
just delete the file if it has no content.

Review commit: https://reviewboard.mozilla.org/r/206062/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206062/

Comment 8

[Gregory Szorc [:gps]]

Attachment: hgmo: use context manager for repo locks;

Mercurial has supported context managers for locks for a while. Let's
use them and make the code cleaner.

Review commit: https://reviewboard.mozilla.org/r/206064/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206064/

Comment 9

[Gregory Szorc [:gps]]

Attachment: hgmo: take the proper lock when writing clonebundles.manifest (bug 1423741);

Mercurial 4.3 issues a developer warning when you attempt I/O on
certain files and the appropriate locks aren't taken. Running the
tests with 4.3 revealed that we were taking the wrong lock when
writing the clonebundles.manifest file. As a refresher, repo.lock()
locks .hg/store and repo.wlock() locks everything in .hg/ except
.hg/store.

Review commit: https://reviewboard.mozilla.org/r/206066/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206066/

Comment 10

[Gregory Szorc [:gps]]

Attachment: hgmo: mark compatible with 4.3 (bug 1423741);

Aside from new test output on 4.3, it looks like the hgmo extensions
"just works." Assuming our test coverage is reasonable anyway.

Review commit: https://reviewboard.mozilla.org/r/206068/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206068/

Comment 11

[Gregory Szorc [:gps]]

Attachment: obsolescencehacks: mark compatible with 4.3 (bug 1423741);

Aside from a test output change, this is pretty straightforward.

Review commit: https://reviewboard.mozilla.org/r/206070/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206070/

Comment 12

[Gregory Szorc [:gps]]

Attachment: pushlog: update test output for 4.3 (bug 1423741);

Add some optional output that appears in 4.3.

Review commit: https://reviewboard.mozilla.org/r/206072/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206072/

Comment 13

[Gregory Szorc [:gps]]

Attachment: pushlog: mark compatibile with 4.3 (bug 1423741);

Mercurial 4.3 (specifically changeset 07439e9f245b) dropped the
obsolete.precursormarkers() function. Fortunately, it was a trivial
function and we can simply use a long-existing API to do the same
thing.

With this change, all pushlog tests pass on 4.3. So we mark the
extensions as compatible.

Review commit: https://reviewboard.mozilla.org/r/206074/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206074/

Comment 14

[Gregory Szorc [:gps]]

Attachment: firefoxtree: add new output from Mercurial 4.3 (bug 1423741);

There is still an issue with bookmarks preventing firefoxtree from
working fully with 4.3. This isn't critical to upgrading hg.mozilla.org.
So I'm going to punt on it.

Review commit: https://reviewboard.mozilla.org/r/206076/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206076/

Comment 15

[Gregory Szorc [:gps]]

Attachment: serverlog: mark as compatible with 4.3 (bug 1423741);

We don't really have tests for serverlog. But if it doesn't work,
it will likely crash the hg.mo server.

Review commit: https://reviewboard.mozilla.org/r/206078/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206078/

Comment 16

[Gregory Szorc [:gps]]

Attachment: hg-web: upgrade Mercurial in hgweb venv to 4.3.3 (bug 1423741);

All the dependent extensions appear to work with 4.3. Let's get the
actual servers running 4.3.

Review commit: https://reviewboard.mozilla.org/r/206080/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206080/

Comment 17

[Gregory Szorc [:gps]]

Attachment: hgtemplates: synchronize templates with Mercurial 4.3.3 (bug 1423741);

We take template content from the 4.3.3 release and add it to
hgtemplates.

Review commit: https://reviewboard.mozilla.org/r/206082/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206082/

Comment 18

[Gregory Szorc [:gps]]

Attachment: hgtemplates: port gitweb theme changes to gitweb_mozilla (bug 1423741);

This commit is a manual port of all gitweb theme changes from 4.2 to 4.3.

Most of the churn is around filelog and annotate views (the mechanism
for line-level annotate was refactored a bit in 4.3).

A CSS class for "footers" in some pages was also changed.

Review commit: https://reviewboard.mozilla.org/r/206084/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206084/

Comment 19

[Gregory Szorc [:gps]]

Attachment: ansible: upgrade Mercurial system package to 4.3.3 (bug 1423741);

See the inline comment for an explanation for why we didn't touch
CentOS 6.

This will update the system Mercurial on hgssh, which we haven't yet
updated to 4.3. However, nothing of importance should be using the
system Mercurial. So it shouldn't matter.

Review commit: https://reviewboard.mozilla.org/r/206086/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206086/

Comment 20

[Gregory Szorc [:gps]]

Attachment: hg-web: use newer hg-experimental revision (bug 1423741);

This revision corresponds to the hg_4.3.1 tag in the repository.

The fastannotate extension switched its config mode. So we update our
hgrc accordingly. There are other changes in fastannotate. But
nothing that looks scary.

setup.py changes also necessitated needing to change our build
invocation in Ansible slightly.

We only have these extensions enabled on some "experimental" repos.
So if this breaks things, it should only break repos that nobody
relies heavily on.

Review commit: https://reviewboard.mozilla.org/r/206088/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/206088/

Comment 21

[Gregory Szorc [:gps]]

Comment on attachment 8935187 [details]
global: mass upgrade to PyYAML 3.12;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206050/diff/1-2/

Comment 22

[Gregory Szorc [:gps]]

Comment on attachment 8935188 [details]
hg-web: use latest version of certifi package;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206052/diff/1-2/

Comment 23

[Gregory Szorc [:gps]]

Comment on attachment 8935189 [details]
hg-web: upgrade mod-wsgi package to latest;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206054/diff/1-2/

Comment 24

[Gregory Szorc [:gps]]

Comment on attachment 8935190 [details]
hg-web: upgrade Mercurial in vcsreplicator environment to 4.3.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206056/diff/1-2/

Comment 25

[Gregory Szorc [:gps]]

Comment on attachment 8935191 [details]
hg-web: upgrade to Mercurial 4.3.3 in tools venv (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206058/diff/1-2/

Comment 26

[Gregory Szorc [:gps]]

Comment on attachment 8935192 [details]
hgmo: add test for behavior when pulling empty manifest;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206060/diff/1-2/

Comment 27

[Gregory Szorc [:gps]]

Comment on attachment 8935193 [details]
hgmo: remove clonebundles.manifest when not present on server;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206062/diff/1-2/

Comment 28

[Gregory Szorc [:gps]]

Comment on attachment 8935194 [details]
hgmo: use context manager for repo locks;

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206064/diff/1-2/

Comment 29

[Gregory Szorc [:gps]]

Comment on attachment 8935195 [details]
hgmo: take the proper lock when writing clonebundles.manifest (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206066/diff/1-2/

Comment 30

[Gregory Szorc [:gps]]

Comment on attachment 8935196 [details]
hgmo: mark compatible with 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206068/diff/1-2/

Comment 31

[Gregory Szorc [:gps]]

Comment on attachment 8935197 [details]
obsolescencehacks: mark compatible with 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206070/diff/1-2/

Comment 32

[Gregory Szorc [:gps]]

Comment on attachment 8935198 [details]
pushlog: update test output for 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206072/diff/1-2/

Comment 33

[Gregory Szorc [:gps]]

Comment on attachment 8935199 [details]
pushlog: mark compatibile with 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206074/diff/1-2/

Comment 34

[Gregory Szorc [:gps]]

Comment on attachment 8935200 [details]
firefoxtree: add new output from Mercurial 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206076/diff/1-2/

Comment 35

[Gregory Szorc [:gps]]

Comment on attachment 8935201 [details]
serverlog: mark as compatible with 4.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206078/diff/1-2/

Comment 36

[Gregory Szorc [:gps]]

Comment on attachment 8935202 [details]
hg-web: upgrade Mercurial in hgweb venv to 4.3.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206080/diff/1-2/

Comment 37

[Gregory Szorc [:gps]]

Comment on attachment 8935203 [details]
hgtemplates: synchronize templates with Mercurial 4.3.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206082/diff/1-2/

Comment 38

[Gregory Szorc [:gps]]

Comment on attachment 8935204 [details]
hgtemplates: port gitweb theme changes to gitweb_mozilla (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206084/diff/1-2/

Comment 39

[Gregory Szorc [:gps]]

Comment on attachment 8935205 [details]
ansible: upgrade Mercurial system package to 4.3.3 (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206086/diff/1-2/

Comment 40

[Gregory Szorc [:gps]]

Comment on attachment 8935206 [details]
hg-web: use newer hg-experimental revision (bug 1423741);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/206088/diff/1-2/

Comment 41

[:glob ✱]

Comment on attachment 8935187 [details]
global: mass upgrade to PyYAML 3.12;

https://reviewboard.mozilla.org/r/206050/#review211660

Comment 42

[:glob ✱]

Comment on attachment 8935188 [details]
hg-web: use latest version of certifi package;

https://reviewboard.mozilla.org/r/206052/#review211662

Comment 43

[:glob ✱]

Comment on attachment 8935189 [details]
hg-web: upgrade mod-wsgi package to latest;

https://reviewboard.mozilla.org/r/206054/#review211664

Comment 44

[:glob ✱]

Comment on attachment 8935190 [details]
hg-web: upgrade Mercurial in vcsreplicator environment to 4.3.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206056/#review211666

Comment 45

[:glob ✱]

Comment on attachment 8935191 [details]
hg-web: upgrade to Mercurial 4.3.3 in tools venv (bug 1423741);

https://reviewboard.mozilla.org/r/206058/#review211668

Comment 46

[:glob ✱]

Comment on attachment 8935192 [details]
hgmo: add test for behavior when pulling empty manifest;

https://reviewboard.mozilla.org/r/206060/#review211670

Comment 47

[:glob ✱]

Comment on attachment 8935193 [details]
hgmo: remove clonebundles.manifest when not present on server;

https://reviewboard.mozilla.org/r/206062/#review211672

Comment 48

[:glob ✱]

Comment on attachment 8935194 [details]
hgmo: use context manager for repo locks;

https://reviewboard.mozilla.org/r/206064/#review211674

Comment 49

[:glob ✱]

Comment on attachment 8935195 [details]
hgmo: take the proper lock when writing clonebundles.manifest (bug 1423741);

https://reviewboard.mozilla.org/r/206066/#review211676

Comment 50

[:glob ✱]

Comment on attachment 8935196 [details]
hgmo: mark compatible with 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206068/#review211686

Comment 51

[:glob ✱]

Comment on attachment 8935197 [details]
obsolescencehacks: mark compatible with 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206070/#review211688

Comment 52

[:glob ✱]

Comment on attachment 8935198 [details]
pushlog: update test output for 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206072/#review211690

Comment 53

[:glob ✱]

Comment on attachment 8935199 [details]
pushlog: mark compatibile with 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206074/#review211692

Comment 54

[:glob ✱]

Comment on attachment 8935200 [details]
firefoxtree: add new output from Mercurial 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206076/#review211694

Comment 55

[:glob ✱]

Comment on attachment 8935201 [details]
serverlog: mark as compatible with 4.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206078/#review211696

heh

Comment 56

[:glob ✱]

Comment on attachment 8935202 [details]
hg-web: upgrade Mercurial in hgweb venv to 4.3.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206080/#review211700

Comment 57

[:glob ✱]

Comment on attachment 8935203 [details]
hgtemplates: synchronize templates with Mercurial 4.3.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206082/#review211702

this one's a rubber-stamp review

Comment 58

[:glob ✱]

Comment on attachment 8935205 [details]
ansible: upgrade Mercurial system package to 4.3.3 (bug 1423741);

https://reviewboard.mozilla.org/r/206086/#review211704

Comment 59

[:glob ✱]

Comment on attachment 8935206 [details]
hg-web: use newer hg-experimental revision (bug 1423741);

https://reviewboard.mozilla.org/r/206088/#review211706

Comment 60

[:glob ✱]

Comment on attachment 8935204 [details]
hgtemplates: port gitweb theme changes to gitweb_mozilla (bug 1423741);

https://reviewboard.mozilla.org/r/206084/#review211710

Comment 61

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6927bbe0bbb3
hg-web: upgrade Mercurial in vcsreplicator environment to 4.3.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6da5fe70957b
hg-web: upgrade to Mercurial 4.3.3 in tools venv ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9e2eb3179031
hgmo: take the proper lock when writing clonebundles.manifest ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/e98a222ff41d
hgmo: mark compatible with 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/1e21435c5fea
obsolescencehacks: mark compatible with 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/6a47167889e7
pushlog: update test output for 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f3d35acfc64d
pushlog: mark compatibile with 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/4481c33d289e
firefoxtree: add new output from Mercurial 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/f2283ea17194
serverlog: mark as compatible with 4.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a70510c2f61d
hg-web: upgrade Mercurial in hgweb venv to 4.3.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a4c244a7cd4f
hgtemplates: synchronize templates with Mercurial 4.3.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/a1983bd9e078
hgtemplates: port gitweb theme changes to gitweb_mozilla ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d1e1bf53e5c0
ansible: upgrade Mercurial system package to 4.3.3 ; r=glob
https://hg.mozilla.org/hgcustom/version-control-tools/rev/0ab3778dfac5
hg-web: use newer hg-experimental revision ; r=glob

Comment 62

[Gregory Szorc [:gps]]

This is deployed.

We still need to upgrade the hgssh servers. Those are a bit more involved. I'll likely get hgweb upgraded to 4.4 before I touch the hgssh servers :/