1308226 - Deploy fastannotate extension to hg.mozilla.org

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

Facebook wrote a "fastannotate" extension (https://bitbucket.org/facebook/hg-experimental/src/8393bd99bb6e0170ecc153d4adbab7e5ba74e9d3/fastannotate/?at=default) that essentially caches annotation data and makes annotate operations much faster. My brief experiments show a 4-10x speedup compared to regular annotate. On files with long history (like https://hg.mozilla.org/mozilla-central/annotate/05d85d9bbce4/layout/base/nsCSSFrameConstructor.cpp - which takes ~20s to load), that can make a huge difference.

If it is suitable for deployment on the server, we should investigate deploying it on hg.mozilla.org so the blame web view is insanely fast.

Because the extension creates extra cache files on disk, we may want to only enable this on certain repos (notably the canonical firefox repos). We may also want to pregenerate annotation caches so first annotate load isn't slow. Although that would result in a lot of extra files on disk. We may want to just let the lazy caching do its job.

Comment 1

[Gregory Szorc [:gps]]

We'll need Mercurial 4.0 on the server to support this. So this won't happen before December.

Comment 2

[Gregory Szorc [:gps]]

As was said IRL at the Mercurial 4.0 Sprint, fastannotate can yield slightly different results than the built-in annotate feature. IIRC, the differences should only come into play after merge commits.

https://bitbucket.org/facebook/hg-experimental/commits/27a94a9c085ad1ee8b1454dde3b83f17d28a66da/raw documents some cases where fastannotate's algorithm differs from the built-in one.

Because of these differences, I was tentatively thinking of only deploying fastannotate on an "experimental" repo initially.

Comment 3

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: clone hg-experimental repo (bug 1308226);

We wish to enable the "fastannotate" extension on hg.mozilla.org.
This extension and its dependencies are in Facebook's "hg-experimental"
repo.

This commit starts the process by installing the hg-experimental repo
on the hgweb machines.

Review commit: https://reviewboard.mozilla.org/r/97004/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/97004/

Comment 4

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: compile and install linelog C extension (bug 1308226);

The fastannotate extension requires the linelog C extension. Compiling
this requires Cython.

Compiling Python extensions requires invoking setup.py. However,
hg-experimental's setup.py defines a number of other C extensions.
While we could hack up setup.py locally to remove, it's slightly
easier to just build them. This required adding a few package
dependencies, unfortunately. But what are you going to do.

We manually copy the built linelog.so file into the hgweb virtualenv.
We don't install the other built extensions.

Review commit: https://reviewboard.mozilla.org/r/97006/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/97006/

Comment 5

[Gregory Szorc [:gps]]

Attachment: ansible/hg-web: install a fastannotate hgrc recipe (bug 1308226);

Rather than repeat the boilerplate code in the .hg/hgrc for each repo
we wish to enable fastannotate on, we can simply "%include" this file.

Review commit: https://reviewboard.mozilla.org/r/97008/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/97008/

Comment 6

[Gregory Szorc [:gps]]

Comment on attachment 8816221 [details]
ansible/hg-web: install a fastannotate hgrc recipe (bug 1308226);

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/97008/diff/1-2/

Comment 7

[Steven MacLeod [:smacleod]]

Comment on attachment 8816219 [details]
ansible/hg-web: clone hg-experimental repo (bug 1308226);

https://reviewboard.mozilla.org/r/97004/#review97222

Comment 8

[Steven MacLeod [:smacleod]]

Comment on attachment 8816220 [details]
ansible/hg-web: compile and install linelog C extension (bug 1308226);

https://reviewboard.mozilla.org/r/97006/#review97224

Comment 9

[Steven MacLeod [:smacleod]]

Comment on attachment 8816221 [details]
ansible/hg-web: install a fastannotate hgrc recipe (bug 1308226);

https://reviewboard.mozilla.org/r/97008/#review97226

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/738b6aaf23ff
ansible/hg-web: clone hg-experimental repo ; r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/9b89fa369c5b
ansible/hg-web: compile and install linelog C extension ; r=smacleod
https://hg.mozilla.org/hgcustom/version-control-tools/rev/376c9195bf11
ansible/hg-web: install a fastannotate hgrc recipe ; r=smacleod

Comment 11

[Gregory Szorc [:gps]]

https://hg.mozilla.org/hgcustom/version-control-tools/rev/d8bd4045cce7af2d9539e317d065ca63f691c100
ansible/hg-web: enable fastannotate on hgweb properly (bug 1308226)

Comment 12

[Gregory Szorc [:gps]]

This is deployed. It has to manually be enabled on repos. I'll send out an announcement (likely tomorrow) saying where this is enabled.