1056981 - Create GPO for Mercurial 3.2.1

Status: RESOLVED FIXED

(Infrastructure & Operations :: RelOps: General, task)

Description

[Chris Cooper [:coop] (he/him)]

Package can be found here:

http://mercurial.selenic.com/release/windows/mercurial-3.1.0-x86.msi

Installation instruction are adapted from here: http://hg.mozilla.org/mozilla-build/file/cfd4e36a9762/packageit.py#l118

msiexec /i mercurial-3.1.0-x86.msi /qn INSTALLDIR=C:\mozilla-build\hg

Builders are the primary target here due to the volume of hg actions, but we should update the Windows testers too.

Comment 1

[Gregory Szorc [:gps]]

I recommend against deploying the initial stable releases (X.Y) of Mercurial in production systems. Instead, I vote for us always waiting on the X.Y.1 or X.Y.2 point releases.

The reason is X.Y releases of Mercurial are major releases and the surface area for bugs and regressions is greater. These are almost always fixed in the X.Y.1 or X.Y.2 releases.

For reference, X.Y releases are made every 3 months. X.Y.Z releases are made every month or as needed (if there is a serious fix).

Look for commits in the "stable" branch of Mercurial and you'll see what I mean regarding fixes. http://selenic.com/repo/hg/shortlog/510cafe72004

Comment 2

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] from comment #1)
> I recommend against deploying the initial stable releases (X.Y) of Mercurial
> in production systems. Instead, I vote for us always waiting on the X.Y.1 or
> X.Y.2 point releases.
> 
> The reason is X.Y releases of Mercurial are major releases and the surface
> area for bugs and regressions is greater. These are almost always fixed in
> the X.Y.1 or X.Y.2 releases.
> 
> For reference, X.Y releases are made every 3 months. X.Y.Z releases are made
> every month or as needed (if there is a serious fix).
> 
> Look for commits in the "stable" branch of Mercurial and you'll see what I
> mean regarding fixes. http://selenic.com/repo/hg/shortlog/510cafe72004

Fair enough.

Corresponding package would then be:

http://mercurial.selenic.com/release/windows/mercurial-3.0.2-x86.msi

Comment 3

[Mark Cornmesser [:markco]]

I have a GPO ready to go for this. How should we roll it out? A small test pool first or to the entire OU?

Comment 4

[Justin Wood (:Callek)]

if I was to make the call I'd say 20 try, 20 non-try builders, and 10-20 of each windows OS of testers.

if that doesn't cause us to rollback within a day, I'd say we rollout to the whole pool.

I'd give coop a chance to say differently, but I'm happy to own the decision in the absence of a competing voice.

Comment 5

[Jordan Lund (:jlund)]

can we hold off on this change for the moment? we are currently trying to capture windows builder regressions. Some of that data is tied to hg_update times.

I want to avoid adding changes into the pot.

We hope to have a clear idea of things by early next week but if this sits as a major priority and can not wait till then, feel free to push back.

Comment 6

[Jordan Lund (:jlund)]

FYI - this is one of the main bugs behind investigation:
'Bug 1055876 - windows build times exceeding timeout on release branches and nonunified trunk branches'

Comment 7

[Gregory Szorc [:gps]]

Please be aware of a performance regression in initial clone with Mercurial 3.0. See bug 1063314.

This *may* not impact `hg unbundle`. You'll have to measure.

Comment 8

[Gregory Szorc [:gps]]

I didn't notice comment #5 when I wrote comment #7.

If "hg_update times" encompasses `hg pull` or `hg unbundle`, I would blame the regression on http://bz.selenic.com/show_bug.cgi?id=4352 / bug 1063314 and I would hold off upgrading past 2.9.2 until my proposed patches land upstream (http://www.selenic.com/pipermail/mercurial-devel/2014-September/061432.html).

If "hg_update times" does not encompass `hg pull` or `hg unbundle`, you should ssh into a machine and run hg commands with `--profile`, paste the results in a bug, and CC me.

Comment 9

[Chris Cooper [:coop] (he/him)]

(In reply to Gregory Szorc [:gps] (away Sep 10 through 27) from comment #8)
> I didn't notice comment #5 when I wrote comment #7.
> 
> If "hg_update times" encompasses `hg pull` or `hg unbundle`, I would blame
> the regression on http://bz.selenic.com/show_bug.cgi?id=4352 / bug 1063314
> and I would hold off upgrading past 2.9.2 until my proposed patches land
> upstream
> (http://www.selenic.com/pipermail/mercurial-devel/2014-September/061432.
> html).
> 
> If "hg_update times" does not encompass `hg pull` or `hg unbundle`, you
> should ssh into a machine and run hg commands with `--profile`, paste the
> results in a bug, and CC me.

Let's not debug hg version issues in our production Windows production pool. If 2.9.2 is safe, let's deploy that for now, and then upgrade to 3.1.2 (or whatever) when the time is right.

Interestingly, there's no msi on the selenic site for 2.9.2, but there are executables:

http://mercurial.selenic.com/release/windows/Mercurial-2.9.2.exe

Mark: does that affect GPO creation at all?

Comment 10

[Mark Cornmesser [:markco]]

> Mark: does that affect GPO creation at all?

It will require a different method but shouldn't be an issue. I will take a look into this later today.

Comment 11

[Mark Cornmesser [:markco]]

I have a GPO ready to roll out for this. How or when should we pull it out?

Comment 12

[Jordan Lund (:jlund)]

ftr - it looks like things are improving and the culprit for Bug 1055876  seems to likely be primarily: https://bugzilla.mozilla.org/show_bug.cgi?id=1062877#c1

landing/progressing this bug shouldn't hurt that investigation effort

Comment 13

[Mark Cornmesser [:markco]]

After taking a deeper look at this I have some questions. 

The GPO i had set up installs the .exe that is provided above, and files end up under C:\program files(x86). Do we actually want to install it as an application or replace the HG files under C:\Mozilla-Build? And If we want to install it as an application what would we like to do with the HG files in C:\Mozilla-Build?

Comment 14

[Gregory Szorc [:gps]]

Mercurial 3.1.2 will fix the revset performance regression documented in comment #8. 3.1.2 should be released within a week. It should be safe to upgrade a day or two after it is released. But 2.9.2 should still be sufficient for the immediate future.

Comment 15

[Mark Cornmesser [:markco]]

coop:
> The GPO i had set up installs the .exe that is provided above, and files end
> up under C:\program files(x86). Do we actually want to install it as an
> application or replace the HG files under C:\Mozilla-Build? And If we want
> to install it as an application what would we like to do with the HG files
> in C:\Mozilla-Build?

Comment 16

[Chris Cooper [:coop] (he/him)]

(In reply to Mark Cornmesser [:markco] from comment #15)
> coop:
> > The GPO i had set up installs the .exe that is provided above, and files end
> > up under C:\program files(x86). Do we actually want to install it as an
> > application or replace the HG files under C:\Mozilla-Build? And If we want
> > to install it as an application what would we like to do with the HG files
> > in C:\Mozilla-Build?

We already munge the PATH to include the C:\mozilla-build\hg\ dir, so we should pave over the existing hg version from MozillaBuild.

Comment 17

[Chris Cooper [:coop] (he/him)]

As gps indicates in https://bugzilla.mozilla.org/show_bug.cgi?id=961279#c21  we can safely upgrade to 3.1.2 now and avoid the performance problems we saw before.

Sorry, Mark, I swear I'm not trying to cause undo churn here. I'd like us to be able to iterate faster on versions of tools to better serve developers, and hg is the standout cross-platform tool example here. hg 3.2 is version with the feature enhancements we *know* we want, so I want to be able to turn around that tool update ASAP once it hits a dot-release (3.2.2 probably).

Comment 18

[Mark Cornmesser [:markco]]

To be safe, is there a small set of machines we push this out to first?

Comment 19

[Mark Cornmesser [:markco]]

(In reply to Mark Cornmesser [:markco] from comment #18)
Coop: 
> To be safe, is there a small set of machines we push this out to first?

Comment 20

[Chris Cooper [:coop] (he/him)]

(In reply to Mark Cornmesser [:markco] from comment #19)
> (In reply to Mark Cornmesser [:markco] from comment #18)
> Coop: 
> > To be safe, is there a small set of machines we push this out to first?

Let's do b-2008-ix-000[1-5] like we did for VS2013.

Comment 21

[Mark Cornmesser [:markco]]

I have disabled 0001, and will use that machine to start working on this.

Comment 22

[Mark Cornmesser [:markco]]

The GPO has been setup and linked the the testing OU. Machines 0001 through 0005 has been moved to the testing OU. 

These machines are still disabled. I did not want to re-enable them and have them burn jobs, if the update goes bad,  without anyone eyes on them.

Comment 23

[Chris Cooper [:coop] (he/him)]

I enabled b-2008-ix-0001 this morning, but quickly had to disable it again. hg operations are failing with the following error:

abort: could not find web.cacerts: C:\Program Files (x86)\Mercurial\cacert.pem

(taken from http://buildbot-master84.srv.releng.scl3.mozilla.com:8001/builders/Firefox%20mozilla-aurora%20win32%20l10n%20nightly/builds/8853/steps/clone_buildtools/logs/stdio)

That Mercurial/ directory doesn't exist.

AFAICT, cacert.pem lives under C:\mozilla-build\hg which is where I though we were install the new package as well.

Comment 24

[Mark Cornmesser [:markco]]

cacert.pem does live that directory. Do we want to change the installation directory for this GPO?

Comment 25

[Chris Cooper [:coop] (he/him)]

(In reply to Mark Cornmesser [:markco] from comment #24)
> cacert.pem does live that directory. Do we want to change the installation
> directory for this GPO?

Are we not already installing to C:\mozilla-build\hg as indicated in comment #0?

Comment 26

[Mark Cornmesser [:markco]]

That is where we are installing to. I was just concerned about:

abort: could not find web.cacerts: C:\Program Files (x86)\Mercurial\cacert.pem

Comment 27

[Justin Wood (:Callek)]

Per chat with Marc today, it _looks_ like this was installing to correct place but a new file not in 2.x (Mercurial.ini) was pointing cacerts at the default location. 

My local `msiexec` to install 3.2.1 adjusted that ini but we were installing this via a file extract instead for some reason.

Comment 28

[Justin Wood (:Callek)]

So to be clear we do want this install to overwrite the Mozilla-build install entirely.

Comment 29

[Mark Cornmesser [:markco]]

A gpo has been set up and has been applied to the test pool. 

This has been set up through an immediate task that will run once using the  3.1.2 .exe with the following arguments: 
/VerySilent 
/LOG="C:\mozilla-build\hg-update.log" 
/DIR="C:\mozilla-build\hg"

Comment 30

[Mark Cornmesser [:markco]]

Correction version 3.2.1. 

And the test pool is is b-2008-ix-0001 through 0005 as mention in comment 20.

Comment 31

[Justin Wood (:Callek)]

so, b-2008-ix-0002 at the least seems to be odd

17:04:37     INFO -  configure:22721: checking for makensisu-3.0a2.exe
17:04:37     INFO -  configure:22787: checking for Unicode NSIS version 2.46 or greater
17:04:37     INFO -  configure: error: To build the installer you must have the latest MozillaBuild or Unicode NSIS version 2.46 or greater in your path.
17:04:37     INFO -  *** Fix above errors and then restart with\
17:04:37     INFO -                 "c:/builds/moz2_slave/m-in-w32-nu-000000000000000000/build/src/mozmake.EXE -f client.mk build"
17:04:37     INFO -  c:/builds/moz2_slave/m-in-w32-nu-000000000000000000/build/src/client.mk:361: recipe for target 'configure' failed
17:04:37     INFO -  mozmake.EXE[2]: *** [configure] Error 1

Are these machines matching production in every other way than hg?

Comment 32

[Justin Wood (:Callek)]

also -0003 got: 
/c/Users/cltbld/.ssh/ffxbld_rsa: No such file or directory

on a fuzzing job

Comment 33

[Justin Wood (:Callek)]

Sooo, coop read those last 2 comments and triggered reimages, YAY

These hosts are running in production, freshly imaged.

HOWEVER they are not using hg 3.2.1 atm, back again on 1.9 so we'll need to re-apply that gpo to these.

Comment 34

[Mark Cornmesser [:markco]]

The re-image was the best bet since those machines had been sitting in the test OU forever. My bad I should had just done it. 

I have moved those machines back to the testing OU. the testing OU inherits all of the build OU GPOs and has the new hg GPO linked to it.

Comment 35

[Justin Wood (:Callek)]

Ok! These 5 are successful, please revert the GPO to prod for these 5, and deploy the mercurial upgrade everywhere.

Comment 36

[Mark Cornmesser [:markco]]

The GPO has been linked to the builder OU and machines have been moved back to their original OU.

Comment 37

[Justin Wood (:Callek)]

This has been live for some time, and no fallout.

Comment 38

[Mark Cornmesser [:markco]]

I will deploy it the larger OUs tomorrow in the am.

Comment 39

[Mark Cornmesser [:markco]]

The GPO has been linked the tester OUs for XP and Win 8.