All users were logged out of Bugzilla on October 13th, 2018

Use a lower zstd compression level for bundles



2 years ago
2 years ago


(Reporter: ted, Unassigned)




(2 attachments)

Created attachment 8844062 [details]
hg clone failure

We're using zstd compression level 22 to generate hg bundles currently:

MozillaBuild on Windows ships with a 32-bit Python, and apparently that compression level is too high and causes decompression to fail (see attachment).

Per discussion on #mercurial, lowering that to level 19 should make it work.
Blocks: 1331084

Comment 1

2 years ago
Unless I'm misreading the zstd source code, level 22 corresponds to a window size of 2^27 or 128 MB. That /should/ fit within a 32-bit address space. It's possible the allocation for a large block of that size failed. I'm going to lower this to 21 to see if 64 MB behaves any better.
Comment hidden (mozreview-request)

Comment 3

2 years ago
Comment on attachment 8850235 [details]
generate-hg-s3-bundles: generate zstd bundles at level 21 (bug 1344790);


i suspect we'll also need to delete the existing zstd-max bundles to force regeneration with the lower level.
Attachment #8850235 - Flags: review?(glob) → review+

Comment 4

2 years ago
Comment on attachment 8850235 [details]
generate-hg-s3-bundles: generate zstd bundles at level 21 (bug 1344790);

Good call. I'll try to remember to do this when I deploy the change.

Comment 5

2 years ago
Pushed by
generate-hg-s3-bundles: generate zstd bundles at level 21 ; r=glob
Last Resolved: 2 years ago
Resolution: --- → FIXED
(In reply to Gregory Szorc [:gps] from comment #4)
> Good call. I'll try to remember to do this when I deploy the change.

Did that happen? I'm still getting an error trying to clone v-c-t on Windows with MozillaBuild Python:
$ ./mach mercurial-setup
Ensuring is up to date at c:/Users/ted/.mozbuild\version-control-tools
*** failed to import extension evolve from c:\build\mutable-history\hgext\ [Errno 2] No such file or directory: 'c:/build/mutable-history/hgext/'
applying clone bundle from
** unknown exception encountered, please report by visiting
** Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)]
zstd.ZstdError: zstd decompress error: Frame requires too much memory for decoding
Error running mach:

Flags: needinfo?(gps)

Comment 7

2 years ago
Yes, it did happen. We only lowered to level 21 though. I thought from the math I did that the window size would be small enough for 32-bit.

Lemme dig into it again and double check the math.

Comment 8

2 years ago
I figured out the problem.

Level 22 corresponds to window log of 27. 21 is 26. And 20 is 25.

2^27 is 128M. That allocation should be possible on 32-bit. But it is a bit large. So we initially dropped the level/window size to 64M. That still didn't work.

What's going on?

Well, in the version of zstd that is bundled with Mercurial 4.1, there is a #define for the max window size. On 32-bit, it is 25. The code in the decompressor explicitly checks this limit and fails decompression without even attempting an allocation. That's why it doesn't level 21 (window 26). So, in order to be compatible with Mercurial 4.1 and 32-bit processes, we'll need to drop the window size to 25, which is level 20.

FWIW, zstd changed the 32-bit max window size limit in 1.1.4. It is now 27. This should make it into Mercurial eventually. But likely not until 4.3 (since python-zstandard still hasn't adopted 1.1.4 and the Mercurial 4.2 freeze is a few days away). I may add something to the clonebundles manifest that identifies the window size of zstd bundles so clients can ignore a bundle with a too-large-for-them window size. That may get into 4.2.

Anyway, we'll need to lower the compression level to 20 on to be compatible with 32-bit Mercurial installs. I'll just push that since it is a trivial change.
Flags: needinfo?(gps)
Resolution: FIXED → ---

Comment 9

2 years ago
Pushed by
generate-hg-s3-bundles: use level 20 for zstd-max
Last Resolved: 2 years ago2 years ago
Resolution: --- → FIXED
Deployed this. Deleted existing zstd-max bundles to make sure they are regenerated if nothing changes and I triggered bundle regeneration so it finished a few hours sooner. mozilla-central's new bundles should be up within an hour or so.
You need to log in before you can comment on or make changes to this bug.