1351513 - Upgrade to Mercurial 4.2 (Windows)

Status: RESOLVED WORKSFORME

(Firefox Build System :: Task Configuration, task)

Description

[Gregory Szorc [:gps]]

+++ This bug was initially created as a clone of Bug #1350437 +++

Let's get TaskCluster Windows workers running Mercurial 4.1.

Comment 1

[Gregory Szorc [:gps]]

Submitted a PR and it was merged and deployed a few hours ago. https://github.com/mozilla-releng/OpenCloudConfig/pull/51

Comment 2

[Gregory Szorc [:gps]]

grenade reverted the PR because of issues like https://treeherder.mozilla.org/logviewer.html#?job_id=87652242&repo=mozilla-beta&lineNumber=163 and were responsible for closing trees.

This issue continues to confound me because we had workers running 4.1 for ~24 hours without issue. I'm not sure what changed to cause this to become an issue.

Comment 4

[Gregory Szorc [:gps]]

4.2 was just released. It contains some fixes to help identify network failures. So we should upgrade straight to 4.2 at this point.

Comment 5

[Bill Gianopoulos [:WG9s]]

This is just me trying to make sense out of things, but would it not be wiser to update the version of Mercurial provided with MozillaBuild to version 4.2 first and then deploy it to taskcluster after getting the kinks out of it?

The version of Mecurial provided with the latest version of MozillaBuild seems to be 3.7.3.

Comment 6

[Gregory Szorc [:gps]]

Upgrading MozillaBuild to Mercurial 4.2 is already in progress.

MozillaBuild ships with 3.7.3 because there hasn't been a MozillaBuild release in a long time and nobody updates MozillaBuild until there is going to be a release.

Comment 7

[Bill Gianopoulos [:WG9s]]

(In reply to Gregory Szorc [:gps] from comment #6)
> Upgrading MozillaBuild to Mercurial 4.2 is already in progress.
> 
> MozillaBuild ships with 3.7.3 because there hasn't been a MozillaBuild
> release in a long time and nobody updates MozillaBuild until there is going
> to be a release.

I realize that but I would think trying to change windows taskcluster builds to a newer release should be a reason to update MozilaBuild if only to include a new Mercurial version it would be nice to work out the bugs on developers builds before using it for customer facing things.  Just my $.02.

Comment 8

[Bill Gianopoulos [:WG9s]]

This situation is bad on two fronts.  we deploy untested things to taskcluster and we end up with things that work on our repos and try that do not build on developers debug platforms.

Comment 9

[Dustin J. Mitchell [:dustin] (he/him)]

Bill, you might want to connect with Jonas -- he's working on the QEMU support we'd need to be able to define task images in-tree for Windows like we currently do for Linux (via Docker). That will be step one in getting to a better place with Windows automation configuration. We would all love some assistance in improving the situation (which is bad on more than two fronts!).

Comment 10

[Bill Gianopoulos [:WG9s]]

OK. If it is something I can help make better rather than just seeming to bitch about it.  I am all over that!

Comment 11

[Bill Gianopoulos [:WG9s]]

(In reply to Dustin J. Mitchell [:dustin] from comment #9)
> Bill, you might want to connect with Jonas -- he's working on the QEMU
> support we'd need to be able to define task images in-tree for Windows like
> we currently do for Linux (via Docker). That will be step one in getting to
> a better place with Windows automation configuration. We would all love some
> assistance in improving the situation (which is bad on more than two
> fronts!).

Oddly one would think this is more important on Windows where the tools are not provided at all by the OS, where they are in Linux (as long as you have developer packages installed) although perhaps either not new enough or too new.

Comment 12

[Dustin J. Mitchell [:dustin] (he/him)]

It's probably more important on windows -- just a lot *easier* on linux.  I think we've now proven the concept that in-tree image definitions are awesome, so it's time to do the hard work.

Comment 13

[Bill Gianopoulos [:WG9s]]

(In reply to Dustin J. Mitchell [:dustin] from comment #12)
> It's probably more important on windows -- just a lot *easier* on linux.  I
> think we've now proven the concept that in-tree image definitions are
> awesome, so it's time to do the hard work.

I sent and email to Jonas volunteering to help.  Not sure what I am getting myself into though!

Comment 14

[Gregory Szorc [:gps]]

(In reply to Bill Gianopoulos [:WG9s] from comment #8)
> This situation is bad on two fronts.  we deploy untested things to
> taskcluster and we end up with things that work on our repos and try that do
> not build on developers debug platforms.

I would rather have automation uncover problems than people because machines are cheaper than developers.

Also, upgrading Mercurial in automation isn't going to break building on developers' machines. I don't see how the 2 are related.

Comment 15

[Bill Gianopoulos [:WG9s]]

(In reply to Gregory Szorc [:gps] from comment #14)
> (In reply to Bill Gianopoulos [:WG9s] from comment #8)
> > This situation is bad on two fronts.  we deploy untested things to
> > taskcluster and we end up with things that work on our repos and try that do
> > not build on developers debug platforms.
> 
> I would rather have automation uncover problems than people because machines
> are cheaper than developers.
> 
> Also, upgrading Mercurial in automation isn't going to break building on
> developers' machines. I don't see how the 2 are related.
The issue is tings will work in automation because they are dependent on newer tools than are what developers are provided. thus making it hard for developers to debug and/or fix.  perhaps mercurial is a bad example of this but the concept is real.

Comment 16

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #15)
> (In reply to Gregory Szorc [:gps] from comment #14)
> > (In reply to Bill Gianopoulos [:WG9s] from comment #8)
> > > This situation is bad on two fronts.  we deploy untested things to
> > > taskcluster and we end up with things that work on our repos and try that do
> > > not build on developers debug platforms.
> > 
> > I would rather have automation uncover problems than people because machines
> > are cheaper than developers.
> > 
> > Also, upgrading Mercurial in automation isn't going to break building on
> > developers' machines. I don't see how the 2 are related.
> The issue is tings will work in automation because they are dependent on
> newer tools than are what developers are provided. thus making it hard for
> developers to debug and/or fix.  perhaps mercurial is a bad example of this
> but the concept is real.

As far as rather picking up in automation.  The issue I was trying to resolve by this suggestion was the long tree clsure cause when mercurial 4.1 was first tried.  The automation is cheaper than developers time, but the issue with taskcluster is that pushing a new build tool is more like putting up a new version on our FTP server it takes a long time for al of the mirrors to pick up the change in the FTP case.  Similarly it takes a long time for the new tool to get deployed to all the build machines in the cluster.  so this results in a timelag between the deployment and when the issue is identified and a long time between when the backout is pushed and that gets to all the build machines.  This also results in downtime for developers because trees are closed for a long time and they can't push anything.

Comment 17

[Bill Gianopoulos [:WG9s]]

(In reply to Bill Gianopoulos [:WG9s] from comment #11)
> (In reply to Dustin J. Mitchell [:dustin] from comment #9)
> > Bill, you might want to connect with Jonas -- he's working on the QEMU
> > support we'd need to be able to define task images in-tree for Windows like
> > we currently do for Linux (via Docker). That will be step one in getting to
> > a better place with Windows automation configuration. We would all love some
> > assistance in improving the situation (which is bad on more than two
> > fronts!).
> 
> Oddly one would think this is more important on Windows where the tools are
> not provided at all by the OS, where they are in Linux (as long as you have
> developer packages installed) although perhaps either not new enough or too
> new.

Can you get me an email for Jonas?  The only Jonas I know is Jonas Sicking who no longer works
 on the project.

Comment 18

[Dustin J. Mitchell [:dustin] (he/him)]

Jonas is at [e-mail]

(In reply to Bill Gianopoulos [:WG9s] from comment #16)
> As far as rather picking up in automation.  The issue I was trying to
> resolve by this suggestion was the long tree clsure cause when mercurial 4.1
> was first tried.  The automation is cheaper than developers time, but the
> issue with taskcluster is that pushing a new build tool is more like putting
> up a new version on our FTP server it takes a long time for al of the
> mirrors to pick up the change in the FTP case.  Similarly it takes a long
> time for the new tool to get deployed to all the build machines in the
> cluster.  so this results in a timelag between the deployment and when the
> issue is identified and a long time between when the backout is pushed and
> that gets to all the build machines.  This also results in downtime for
> developers because trees are closed for a long time and they can't push
> anything.

This is a problem that in-tree images will address.  Docker provides the proof of concept: when we make a change on Linux (such as upgrading Mercurial), we can modify the in-tree Dockerfile and push to try.  Assuming it succeeds, the resulting landing rides the trains just like any other source-code change.  If an issue appears after some time, we can uplift a backout the affected branches.

We're not there yet -- OCC works more like Puppet/GPO, which is what configures Buildbot workers, than like Docker.

Comment 19

[Justin Wood (:Callek)]

:gps, is this still a thing?

Comment 20

[Gregory Szorc [:gps]]

I just poked through OpenCloudConfig and it appears we're using Mercurial 4.3.3 everywhere.

So unless there are some ancient buildbot workers around or other Windows machines not managed by OCC, this looks to be done. I'll let someone else with more knowledge close the bug.

Comment 21

[Gregory Szorc [:gps]]

OpenCloudConfig is using Mercurial 4.5 these days.

Not sure what Puppet/GPO are using. But a specific bug for Mercurial 4.2 is old and jaded.