1217817 - Ship watchman and hgwatchman in MozillaBuild

Status: RESOLVED FIXED

(Firefox Build System :: MozillaBuild, task)

Description

[Gregory Szorc [:gps]]

hgwatchman makes `hg status` and other commands that walk the working copy much faster. It relies on the Watchman file watching tool, which until recently did not support Windows.

The author of Watchman is standing over my shoulder. Windows support is in "alpha." But that label is only because they don't have any users at scale to call it "beta" or better.

We should:

1) Install watchman in MozillaBuild
2) Ship hgwatchman in MozillaBuild (or vendor in version-control-tools)
3) Enable hgwatchman by default in MozillaBuild's global hgrc

We can find a link to the latest watchman Windows build at https://ci.appveyor.com/project/wez/watchman/branch/master/artifacts. More info at https://github.com/facebook/watchman/issues/19.

The way I see it, we can ship this enabled in a beta of MozillaBuild. If watchman blows up, we just don't enable hgwatchman in the global hgrc.

Comment 1

[Xidorn Quan [:xidorn] UTC+11]

Good to see this. I thought Windows had a good cache so that even if the repo is large, `hg status` won't be too slow. But it seems that's not true. The command becomes much slower than it was when I started using this Windows machine.

Comment 2

[Ryan VanderMeulen [:RyanVM]]

I'm assuming we need pywatchman as well per https://github.com/facebook/watchman/issues/19 ?

Comment 3

[Ryan VanderMeulen [:RyanVM]]

Looks like the hgwatchman package on pypi includes it, so nevermind.

Comment 4

[Ryan VanderMeulen [:RyanVM]]

I'm not really sure how to test this, but here's a build that I believe has everything setup correctly for it. Can you try it out, Greg?

http://people.mozilla.org/~rvandermeulen/MozillaBuildSetup2.2.0pre1.exe (77.5MB)
SHA-256: f929bffff830d608a81206592458df1bc9a40db4627649f0c8e9d4d8c0e1d2fe

Comment 5

[Gregory Szorc [:gps]]

I don't have access to a Windows machine this week. But if you want to test this yourself, try running `hg status` a few times. With hgwatchman installed, it should complete in a fraction of the time as without hgwatchman. You can disable watchman on the command invocation via `hg --config watchman.mode=off status`.

Comment 6

[Gregory Szorc [:gps]]

watchman + hgwatchman appears to be working in this build! `hg status` goes from ~1.680s to ~0.280s on my machine. Not bad. `git status` (from Git for Windows Win32 distribution) is ~0.363s by comparison, making hg the winner.

FWIW, `hg --version` is taking ~0.157s, which feels a bit high. I may have to perform some performance profiling to see what's up.

Comment 7

[Gregory Szorc [:gps]]

Disabling all extensions causes `hg --version` to run in ~0.049s. Looks like it's time to refactor some code in version-control-tools to optimize for load time.

Comment 8

[Xidorn Quan [:xidorn] UTC+11]

I wonder whether it would make sense to use pypy+chg to optimize the performance. (Although it seems chg is not available on Windows, so it is not actually relevant to this bug.)

Comment 9

[Ryan VanderMeulen [:RyanVM]]

http://hg.mozilla.org/mozilla-build/rev/b092537e51b2
http://hg.mozilla.org/mozilla-build/rev/d2b21afc1c0c
http://hg.mozilla.org/mozilla-build/rev/9a412956ed10

This enables it by default for now. Will be trivial to turn off if needed, though I'm not sure how we're going to judge whether that's the case or not.