1336190 - Generate zstd bundles by default

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, defect)

Description

[Gregory Szorc [:gps]]

In bug 1331084 we added support for generating zstd bundles. zstd bundle generation is opt in and we only enabled it for specific repos.

It looks like the zstd bundles we've deployed "just work," so I think it is safe to enable zstd bundle generation by default.

The end state is we'll always generate zstd (level 3) bundles and will have a per-repo option in the manifest to enable slow/maximum zstd (level 22) compression.

Comment 1

[Gregory Szorc [:gps]]

Attachment: hgserver: generate zstd bundles by default (bug 1336190);

zstd-max bundles for mozilla-central, mozilla-unified, and other repos
appear to be working just fine.

So let's enable zstd bundle generation on all repos by default.

zstd bundles will be preferred over other bundles (unless in an EC2
region, which continues to favor stream clone bundles).

If the zstd_max option is present, we will generate zstd bundles
with maximum compression instead of the default compression.

Review commit: https://reviewboard.mozilla.org/r/109814/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/109814/

Comment 2

[:glob ✱]

what impact will this have on the load on the servers, if any?

Comment 3

[Gregory Szorc [:gps]]

There is a systemd timer that runs on the hgssh master server when the sun is over the pacific ocean (read: during low load time) that generates and uploads these bundles. It runs as a low priority process so actual server traffic takes precedence. Bundles are only generated if there is new data (so old repos like ESRs often don't result in new generation). At most 4 bundles are generated concurrently. The server has 8 physical cores IIRC and are generally pretty idle.

zstd level 3 bundles (what we're enabling) for Firefox repos will take 2-3.5 minutes of CPU time to generate. CPU capacity should not be a problem.

Generally speaking, this bundle generation is I/O bound because of NFS. What I've observed is that NFS tends to hold up progress on all bundles and all concurrently generated bundles end up accessing the same files at the same time because NFS can't manage to serve files as fast as bundle generation can consume them. This basically means we can add more bundles/CPU and unless bundle generation is really slow (read zstd level 22 or bzip2), they'll complete at roughly the same rate and won't cause too much overhead.

tl;dr I'm not too worried about capacity. If it turns into an issue, it is trivial to stop the process on the hgssh master, trigger another time, or revent this change and wait for the next bundle generation.

Comment 4

[:glob ✱]

(In reply to Gregory Szorc [:gps] (disappearing for a month after 2017-02-10) from comment #3)
> The server has 8 physical cores IIRC and are generally pretty idle.

i've been looking at the servers a bit recently there's usually at least one process pegging one cpu to 100% (normally hg-aggregate).  with that mount of cores it shouldn't matter, but i don't agree this box is generally idle.

it's a shame graphite doesn't appear to be monitoring the load on this server :(

is there a reason bundle generate has run on the hgssh master (instead of the mostly idle slave)?

Comment 5

[Gregory Szorc [:gps]]

hg-aggregate is the user/process that periodically `hg pull`s various repos into other repos. That is what populates mozilla-central-gd and mozilla-unified. Right now it just runs in a loop: it needs to be changed to look up Kafka events and only run when necessary. I think there is a bug on that. Outside of that, the machine is idle except for `hg push` traffic and bundle generation. We still have plenty of CPU capacity.

Bundle generation could run on the mostly idle spare. It was easier to have the systemd service hooked into `hg-master.target` because, well, the point of the master/spare division is that one has everything and the other has nothing. We could change this, of course. Would need a separate file on the NFS mount to control which machine should be running the timer/service.

Comment 6

[:glob ✱]

Comment on attachment 8833587 [details]
hgserver: generate zstd bundles by default (bug 1336190);

https://reviewboard.mozilla.org/r/109814/#review111364

thanks for answering my capacity related questions.
this looks good for deploying on the hg master.
please provide clear instructions on how to disable should the need arise.

Comment 7

[Gregory Szorc [:gps]]

Comment on attachment 8833587 [details]
hgserver: generate zstd bundles by default (bug 1336190);

https://reviewboard.mozilla.org/r/109814/#review111364

Well, we can't disable bundle generation completely or else the servers melt from too much load. But first, automation will start failing clones because the bundles referenced by the bundle manifests will have expired in S3 and will HTTP 404.

So if this change blows up the server, the recourse is to back it out and deploy the old version. I'll deploy this tomorrow morning and kick off a manual bundle generation during the day. That should be a good stress test.

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/781a7a1a1e16
hgserver: generate zstd bundles by default ; r=glob

Comment 9

[Gregory Szorc [:gps]]

FWIW, zstd 1.1.3 has experimental support for multi-threaded compression. I'm not sure if the output is compatible with legacy clients or what. But if we turn that on (which I'd like to investigate because it is much faster), we could definitely run into CPU exhaustion issues depending on how many compression threads we use. But that isn't going to happen before Friday :)