992368 - Create process for upgrading Mercurial on a schedule

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Ben Kero [:bkero]]

There is a demand for new versions of Mercurial on the hg.mozilla.org infrastructure due to stability, performance, and new features.

It has been demonstrated that the server and client versions of Mercurial need not be the same. During operations both sides will negotiate capabilities that they both support.

To accomplish this we'll need to define a schedule this operates on. Recommendations include following the Mercurial release schedule, which is every 3 months. This is long enough for new versions to stabliize and short enough that we shouldn't run into upgrade issues once the process has been determined.

We'll also need a set of QA requirements from Release Engineering that we can automate to minimize the amount of manual QA work that needs to be done for each release.

IT already has the capabilities to roll our own RPMs for new versions, and has done so for all recent Mercurial upgrades.

Comment 1

[Gregory Szorc [:gps]]

Mercurial's release schedule is at http://mercurial.selenic.com/wiki/TimeBasedReleasePlan

tl;dr major releases Feb, May, Aug, and Nov 1. Minor (typically bugfix and minor issue only) releases every month.

Given the rapid rate of development on Mercurial as of late and the efficiency gains and bug fixes those entail, I second shadowing the 3 month releases. If we want to mitigate risk, we would deploy the 1st or 2nd minor release after each major version. That limits our exposure to bugs introduced in major releases and lets others do most of the testing for us.

mpm: I'm curious if you have any recommendations for server operators. Currently Mozilla is running 2.5.4.

Comment 2

[Matt Mackall]

Facebook internally deploys Mercurial's development tip to servers on a roughly weekly basis with basically no disruption. So I think you'll be just fine with just about any upgrade schedule. Just don't deploy on Friday evening.

Comment 3

[Ben Kero [:bkero]]

The work that's gone into version-control-tools (especially create-test-environment and run-mercurial-tests.py) in addition to the documentation about building and deploying package updates has taken us much closer to updating on a regular cycle.

Is this enough to mark this as done, or should we create some more material relating to the scheduling process for how the upgrades should take place?

Comment 4

[hwine]

We have the pieces now, we just need the actual steps and order to stitch it together. E.g. something like:
 - add Hg version X to test harness (see XXXXX)
 - get the following test suites to pass:
   - run-mercurial-tests.py ...
   - ...
 - build packages
 - deploy to staging
 - run tests YYYY
 - schedule cutover time with sheriffs
 - cutover & run post cutover validations
   - push commit to m-i (e.g. this would have caught last TCW issue)
   - verify with sheriffs that system look good
 - declare victory

That will give us a place to keep "lessons learned" (probably in the form of additional post cutover checks) as we proceed. While the traditional place for this is in mana, it may make sense to have the several documented in a public place, as community members could perform those steps. (I'd be for having the entire checklist public, with just a pointer in mana, but will defer to group decision.)

Greg - would version-control-tools/docs/moz-upgrade.rst be a reasonable spot for the checklist?

Comment 5

[Gregory Szorc [:gps]]

Yeah, throwing some docs under /docs/ sounds like a plan.

I was thinking we may want to create an entire directory/section for hg.mozilla.org/git.mozilla.org/etc implementation and admin notes.

Comment 6

[Ben Kero [:bkero]]

Attachment: Mercurial upgrade instructions

I've attached a preliminary version of a document for how to upgrade Mercurial safely and successfully. Let me know what you think.

Comment 7

[Ed Morley [:emorley]]

Comment on attachment 8533928 [details]
Mercurial upgrade instructions

>Webheads
>^^^^^^^^
>
...
>Repeat this procedure until all webheads have been upgraded.

The step to change each node back from "draining" to active is missing - following the steps as is will leave zero webheads active :-)

Comment 8

[hwine]

Comment on attachment 8533928 [details]
Mercurial upgrade instructions

Nice!

A couple of questions I'd like to see answered - only the first one is a blocker:

1. I don't see an explanation for "Re-add and confirm correctness".

2. The yum process as explained to be was to always use "yum-wrapper" if it exists.

3. For the coordination step, advance notification via CAB is preferred - what you have is minimum for an emergency up/down grade. (This can be expanded later.)

Comment 9

[Gregory Szorc [:gps]]

Please indent code blocks like so:

.. code-block:: shell

   $ command

Generally 3 space indent is used in RST.

For the testing against a specific hg, you don't need to hack up test-requirements.txt. Instead:

   $ ./run-mercurial-tests --with-hg /path/to/hg

We should probably automated RPM building as part of the Jenkins job...

Comment 10

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/273ab70b567f

Comment 11

[Gregory Szorc [:gps]]

Comment on attachment 8533928 [details]
Mercurial upgrade instructions

I went ahead and pushed this (attributed to bkero) into v-c-t. I made some minor formatting changes to make things play nicer with Sphinx.

Feel free to make any subsequent changes directly in source control.

Comment 12

[hwine]

ni :bkero - once comment 8 is addressed, this is ready to resolve fixed

Comment 13

[Ben Kero [:bkero]]

Attachment: MozReview Request: bz://992368/bkero

/r/6703 - Bug 992368: docs: update server documentation

Pull down this commit:

hg pull -r 9140d9450ef4d25318b45495f18e6f516ff08960 https://reviewboard-hg.mozilla.org/version-control-tools/

Comment 14

[hwine]

Comment on attachment 8589355 [details]
MozReview Request: bz://992368/bkero

https://reviewboard.mozilla.org/r/6701/#review5643

lgtm - thanks!

Comment 15

[Gregory Szorc [:gps]]

Comment on attachment 8589355 [details]
MozReview Request: bz://992368/bkero

https://reviewboard.mozilla.org/r/6701/#review5645

Ship It!

Comment 16

[Ben Kero [:bkero]]

Merged in

http://hg.mozilla.org/hgcustom/version-control-tools/rev/b1d9e9814cde

Comment 18

[Ben Kero [:bkero]]

Attachment: MozReview Request: Bug 992368: docs: update server documentation