1344790 - Use a lower zstd compression level for hg.mo bundles

Status: RESOLVED FIXED

(Developer Services :: Mercurial: hg.mozilla.org, enhancement)

Description

[(not currently active) Ted Mielczarek]

Attachment: hg clone failure

We're using zstd compression level 22 to generate hg bundles currently:
https://hg.mozilla.org/hgcustom/version-control-tools/file/781a7a1a1e16/scripts/generate-hg-s3-bundles#l37

MozillaBuild on Windows ships with a 32-bit Python, and apparently that compression level is too high and causes decompression to fail (see attachment).

Per discussion on #mercurial, lowering that to level 19 should make it work.

Comment 1

[Gregory Szorc [:gps]]

Unless I'm misreading the zstd source code, level 22 corresponds to a window size of 2^27 or 128 MB. That /should/ fit within a 32-bit address space. It's possible the allocation for a large block of that size failed. I'm going to lower this to 21 to see if 64 MB behaves any better.

Comment 2

[Gregory Szorc [:gps]]

Attachment: generate-hg-s3-bundles: generate zstd bundles at level 21 (bug 1344790);

Level 22 was generating an out of memory error on 32-bit Windows.
Let's try level 21, which has a window size of half of level 22.
This will make the bundles a little larger. But, that's the price
we may have to pay for 32-bit compatbility.

Review commit: https://reviewboard.mozilla.org/r/122882/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/122882/

Comment 3

[:glob ✱]

Comment on attachment 8850235 [details]
generate-hg-s3-bundles: generate zstd bundles at level 21 (bug 1344790);

https://reviewboard.mozilla.org/r/122882/#review125200

r=glob

i suspect we'll also need to delete the existing zstd-max bundles to force regeneration with the lower level.

Comment 4

[Gregory Szorc [:gps]]

Comment on attachment 8850235 [details]
generate-hg-s3-bundles: generate zstd bundles at level 21 (bug 1344790);

https://reviewboard.mozilla.org/r/122882/#review125200

Good call. I'll try to remember to do this when I deploy the change.

Comment 5

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/d69c39abeced
generate-hg-s3-bundles: generate zstd bundles at level 21 ; r=glob

Comment 6

[(not currently active) Ted Mielczarek]

(In reply to Gregory Szorc [:gps] from comment #4)
> Good call. I'll try to remember to do this when I deploy the change.

Did that happen? I'm still getting an error trying to clone v-c-t on Windows with MozillaBuild Python:
$ ./mach mercurial-setup
================================================================================
Ensuring https://hg.mozilla.org/hgcustom/version-control-tools is up to date at c:/Users/ted/.mozbuild\version-control-tools
*** failed to import extension evolve from c:\build\mutable-history\hgext\evolve.py: [Errno 2] No such file or directory: 'c:/build/mutable-history/hgext/evolve.py'
applying clone bundle from https://hg.cdn.mozilla.net/hgcustom/version-control-tools/60d4ae1f28082b058830d24cf7cebcace5b43915.zstd-max.hg
** unknown exception encountered, please report by visiting
** https://mercurial-scm.org/wiki/BugTracker
** Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]
<...>
zstd.ZstdError: zstd decompress error: Frame requires too much memory for decoding
================================================================================
Error running mach:

    ['mercurial-setup']
<...>

Comment 7

[Gregory Szorc [:gps]]

Yes, it did happen. We only lowered to level 21 though. I thought from the math I did that the window size would be small enough for 32-bit.

Lemme dig into it again and double check the math.

Comment 8

[Gregory Szorc [:gps]]

I figured out the problem.

Level 22 corresponds to window log of 27. 21 is 26. And 20 is 25.

2^27 is 128M. That allocation should be possible on 32-bit. But it is a bit large. So we initially dropped the level/window size to 64M. That still didn't work.

What's going on?

Well, in the version of zstd that is bundled with Mercurial 4.1, there is a #define for the max window size. On 32-bit, it is 25. The code in the decompressor explicitly checks this limit and fails decompression without even attempting an allocation. That's why it doesn't level 21 (window 26). So, in order to be compatible with Mercurial 4.1 and 32-bit processes, we'll need to drop the window size to 25, which is level 20.

FWIW, zstd changed the 32-bit max window size limit in 1.1.4. It is now 27. This should make it into Mercurial eventually. But likely not until 4.3 (since python-zstandard still hasn't adopted 1.1.4 and the Mercurial 4.2 freeze is a few days away). I may add something to the clonebundles manifest that identifies the window size of zstd bundles so clients can ignore a bundle with a too-large-for-them window size. That may get into 4.2.

Anyway, we'll need to lower the compression level to 20 on hg.mozilla.org to be compatible with 32-bit Mercurial installs. I'll just push that since it is a trivial change.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/hgcustom/version-control-tools/rev/3cc64275a536
generate-hg-s3-bundles: use level 20 for zstd-max

Comment 10

[Gregory Szorc [:gps]]

Deployed this. Deleted existing zstd-max bundles to make sure they are regenerated if nothing changes and I triggered bundle regeneration so it finished a few hours sooner. mozilla-central's new bundles should be up within an hour or so.