1370419 - Serve stream clones to Mozilla offices

Status: RESOLVED DUPLICATE of bug 1370420

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[Gregory Szorc [:gps]]

botond pinged me on IRC today and said Firefox clones were really slow. Upon investigation, I think we turned a corner with the number of files in the repo that is causing excessive slowdown on Windows.

As https://docs.google.com/spreadsheets/d/1wja68eIZoiP-dOb2yvzJYUXeKr8MzEpPFJ3WjSBldiE/edit#gid=0 shows, repo size has increased significantly in the past few months. This slows down Mercurial clones/pulls for the following reasons:

1) Mercurial does I/O with a single thread
2) Mass file creation is a bit slow on Windows/NTFS
3) Windows Defender makes file creation even slower due to scanning changed files at close time

The impact of Windows Defender is significant. Here are `hg unbundle` times for a zstd bundle:

Defender enabled:  27:32 
Defender disabled: 12:29

Defender more than doubles `hg unbundle` time when enabled. Yikes!

That 12:29 still isn't great, especially since Linux can do it several minutes faster on the same machine. But let's not go there right now.

Mercurial has an alternate "stream clone" mechanism that results in more efficient clone times. Typically when Mercurial clones, the client decompresses a "bundle" and then recalculates things on the client, such as compressed deltas. On repos the size of ours this adds several minutes to clone times. In cases where the client is newer than server storage, the client can transparently upgrade to modern storage formats, etc. So that overhead is acceptable and is a useful abstraction layer. But for cases where the server and client are in agreement, it doesn't buy you much.

The major downside to "stream clone" is that the clone size is larger. Much larger. With traditional clones, all data is decompressed then fed into a single compressor. With large compression window sizes, data from discrete files compresses well with each other. In "stream clone" mode, you slurp files off disk which are independently compressed, not getting benefits from shared compression windows. This is roughly equivalent to tar.gz versus zip file formats. Currently, a "stream clone" of mozilla-unified is 1,145 MB larger (1,974 vs 829 MB). That's a massive different, especially for people on slower Internet connections. So it makes sense to not use stream clones by default because it will be a *really* bad experience for someone on a slow connection.

Furthermore, "stream clones" have an optimization that can work around Windows Defender overhead. Here are times for `hg debugapplystreamclone`:

Defender enabled:  4:45
Defender disabled: 2:52
Defender enabled / hg optimization disabled: 17:28

The takeaway here is that stream clones with default Windows and Mercurial configs are substantially faster: 4:45 vs 27:32. And if you have Defender disabled, the difference is 2:52 vs 12:29. Assuming the server's storage format is optimal (which it is for the Firefox repos after partial completion of bug 1351859), the only significant downside to stream clones is transferring extra data - currently 1,145 MB - over the network.

Given the drastic difference in clone times, I think we should serve stream clones to Mozilla office IPs by default if we know the office has a decent Internet connection (one would hope they all do). We already do this for AWS client IPs. So all we need are public IP blocks for Mozilla offices and we can reconfigure hg.mozilla.org so clones from Mozilla offices are several minutes faster with no cooperation from the client needed. This will have an immediate and drastic impact on clone performance and will make a critical part of Firefox development on Windows suck much, much less.

arr: could you please help me acquire the public IP blocks of Mozilla offices? Also, do the blocks change frequently? Is there a change process we can hook into so we know when to update the server config? Alternatively, Puppet or some other process could install a file that the server reads. All we need is IP blocks in a machine readable format and we're good to go.

Comment 1

[Ed Morley [:emorley]]

Double post.