Closed Bug 1880009 Opened 2 years ago Closed 1 year ago

use lzma for firefox dmgs

Categories

(Firefox Build System :: General, defect, P3)

Product:
Firefox Build System
Component:
General
Version:
Firefox 123
Platform:
All
macOS
Type:
defect

Tracking

(firefox135 fixed)

Status:
RESOLVED FIXED
Milestone:
135 Branch
Tracking Flags:
Tracking Status
firefox135 --- fixed

People

(Reporter: arthur200126, Assigned: vasi)

References

Details

Attachments

(1 file, 3 external files)

Screenshot 2024-02-13 at 14.11.24.png
288.71 KB, image/png
Details

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/17.4 Safari/605.1.15

Steps to reproduce:

  1. Download latest dmg
  2. Open, drag app to Applications

Actual results:

Install takes 4 minutes on m3 CPU (MacBook 12 inch Retina, 2017)

Expected results:

Install should have been much faster.

Using "file" on the dmg shows that it was compressed by bzip2, which explains why it is so slow. I have previously filed a very similar bug at GIMP: https://gitlab.gnome.org/GNOME/gimp/-/issues/9032.

Using lzfse dmg requires macOS 10.11. This is not a problem, because Firefox requires 10.15 minimum.

OS: Unspecified → macOS
Hardware: Unspecified → All

I don't know what the dmg-packaing component is called, so I can't change quite the component here. It probably affects multiple products -- old dmg packaging code do not tend to get updated.

Component: Untriaged → General
Product: Firefox → Firefox Build System

GitHub org-wise search gives a bunch of potential hits. I recommend changing all the UDBZ (bzip2) to ULFO. The former is more severe in terms of speed penalty.
https://github.com/search?q=org%3Amozilla+UDBZ+hdiutil&type=code

ULFO might have reduced compression ratio; it also does not have a level switch. Test before you ship!

Component: General → Android Studio and Gradle Integration
Summary: [macOS] dmg with bzip2 compression too slow to install; please use lzfse → [macOS, mozpack] dmg with bzip2 compression too slow to install; please use lzfse

Actually, since the minimum is 10.15, you can also get away with ULMO, which uses lzma. It's hopefully still faster than bzip2 and probably a little stronger.

See the more current man page at https://keith.github.io/xcode-man-pages/hdiutil.1.html

Here's the comparison between the formats, size of dmg, time taken by mount and copy (average of 10 runs, on Mac Studio 2022), with the Nightly 2024-02-13 universal binary dmg.
UDBZ is the current one.

format imagekey dmg size [bytes] mount [s] copy [s]
UDBZ bzip2-level=9 155,633,689 9.862 10.503
UDCO - 210,296,997 0.874 0.848
UDZO zlib-level=9 170,201,922 1.489 1.440
ULFO - 167,605,978 0.571 0.545
ULMO - 130,746,420 6.139 6.368

ULMO becomes the smallest. it also reduces the time taken by mount and copy by ~40%.
ULFO has small increase, but it significantly reduces the time taken by mount and copy, by ~95%.

Component: Android Studio and Gradle Integration → General

Thanks for the benchmark -- the relative speeds and sizes make sense. I notice that your mount time is almost the same as copy time, which also makes sense considering hdiutil by default verifies the dmg's checksum.

Oops, didn't realize that I fumbled with Component.

I've tried a converted ULMO on my end. Finder copy takes about 40 seconds. Command line mount takes 19, follow-up copy takes 14; could be a fluke.

The dmg file is generated in 2 places in tree, one is for linux, and one is for macOS. The linux's path uses https://github.com/mozilla/libdmg-hfsplus , and macOS case uses system-builtin hdiutil command.

The release dmg file is generated on linux on automation, and libdmg-hfsplus supports only bzip2 format as an output file.

So, in order to switch to ULFO(lzfse) or ULMO(lzma), that format needs to be implemented in libdmg-hfsplus.

(In reply to Tooru Fujisawa [:arai] from comment #6)

The dmg file is generated in 2 places in tree, one is for linux, and one is for macOS. The linux's path uses https://github.com/mozilla/libdmg-hfsplus , and macOS case uses system-builtin hdiutil command.

The release dmg file is generated on linux on automation, and libdmg-hfsplus supports only bzip2 format as an output file.

Just to add some context to this:

  • We are tied to Linux for mac packaging because we can't scale mac hardware machines well enough to make it practical to package on them (this is what pushed us to libdmg-hfsplus in the first place, as I understand it).
  • The recently completed work on macOS attribution means we must package with libdmg-hfsplus regardless of the host OS we package on, unless we make the call to live without that feature.

Oops, that's going to be a bit of a pickle. I see libdmg-hfsplus knows nothing about LZFSE or LZMA (actually xz if you run "file"). The repo at https://github.com/matthewbauer/undmg might be useful for LZFSE stuff -- at least the dmg.h is similar.

Severity: -- → S3
Priority: -- → P3

I skimmed undmg a little more at. Turns out it's basically a very cut-down libdmg with clang-format run through (I get it: the indents are not great), so its changes are going to be very applicable here. The LZFSE bit boils down to feeding the entire block to lzfse_decode_buffer, which makes sense considering zip and bzip are done similarly. I guess compression support would just be feeding the whole thing to lzfse_encode_buffer. There are no options in LZFSE, so it's unlikely we create something that real macs can't decode.

"LZMA" (xz) is presumably the same deal. The BLOCK_LZMA constant is 0x80000008, based on sequential guessing and confirmed by https://github.com/libyal/libmodi/blob/main/documentation/Mac%20OS%20disk%20image%20types.asciidoc. Libmodi, which is decode-only, just makes calls to liblzma (XZ Util) for decode. Now xz does have a flurry of fancy options; the Apple Compression Framework is known to only offer LZMA level 6, presumably single-threaded (all in one chunk), presumably without BCJ -- that can be replicated by just calling lzma_stream_encoder with defaults. I've no idea if it supports decompressing anything else.

This should be reasonably straightforward, and I intend to try it. I'm the one who built bzip2 support waaaay back in the day, and I've also added LZFSE support in my own branch of libdmg: https://github.com/vasi/libdmg-hfsplus/commit/71b22c724cddcd21b3048da98154276430619230

Btw, do we have any numbers on how much Mozilla spends on macOS Firefox downloads, or at least how many macOS Firefox downloads there are? It'd be great to be able to quantify what this saves!

https://bugzilla.mozilla.org/show_bug.cgi?id=1928742 is picking up the new libdmg-hfsplus with support for lzma & lzfse (thanks Dave!!!). Once that lands and sticks, we can start thinking about enabling different compression.

I don't know yet whether we'll want lzfse or lzma. Dave's data shows that lzfse would greatly reduce decompression time, while lzma would greatly reduce size. It's not immediately obvious to me which we would want to choose, seeing as both are in the critical path of installation.

Depends on: 1928742

Dave, I'm assigning this to you because you've done all of the work to make this happen, and you deserve the credit. When we flip this on, we'll need some minor changes to the Gecko tree to enable it. You're welcome to do that if you like, but I'm also happy to do it myself regardless of the bug assignment.

Assignee: nobody → dave

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #13)

Dave, I'm assigning this to you because you've done all of the work to make this happen, and you deserve the credit. When we flip this on, we'll need some minor changes to the Gecko tree to enable it. You're welcome to do that if you like, but I'm also happy to do it myself regardless of the bug assignment.

Looks like we also need to update https://searchfox.org/mozilla-central/rev/dfaf02d68a7cb018b6cad7e189f450352e2cde04/taskcluster/gecko_taskgraph/transforms/repackage.py#200-212 for dmg build invocations.

When we flip this on, we'll need some minor changes to the Gecko tree to enable it

It looks like testing this locally will require me to cross-build macOS Firefox on Linux. Are there docs on setting this up anywhere? I could probably chase all the required toolchains, sysroots, etc all over the build infra, but if it's written down somewhere that'd be easier. cf https://bugzilla.mozilla.org/show_bug.cgi?id=921040

(In reply to Dave Vasilevsky from comment #15)

When we flip this on, we'll need some minor changes to the Gecko tree to enable it

It looks like testing this locally will require me to cross-build macOS Firefox on Linux. Are there docs on setting this up anywhere? I could probably chase all the required toolchains, sysroots, etc all over the build infra, but if it's written down somewhere that'd be easier. cf https://bugzilla.mozilla.org/show_bug.cgi?id=921040

To be honest, I haven't done a cross build for mac locally in ages. It would definitely be one form of verification, but ultimately the DMGs we ship come from CI tasks that wouldn't use the same code. Unless it looks like a lot of iteration is going to be required, it might be easier to have me send things through CI for testing, or to request access to do so yourself (which I will happily vouch for).

Let's go with lzma level 5 for now; we can tweak in the future if we need want to.

We're also going to hold this on Nightly for a period of time. This will require a bit of extra wiring on the CI side; I haven't quite sorted out the best way to do that yet. I may follow up with that after we get it turned on in the first place.

ac_add_options --target=aarch64-apple-darwin

Configure will tell you to run a rustup command, but that's all you'll need.

Thanks :glandium!

I applied to Level 1 access here: https://bugzilla.mozilla.org/show_bug.cgi?id=1933523

Let's go with lzma level 5 for now; we can tweak in the future if we need want to.

Great, that'll be easier than lzfse, since Debian 8 definitely doesn't have liblzfse.

Depends on: 1933523
Summary: [macOS, mozpack] dmg with bzip2 compression too slow to install; please use lzfse → use lzma for firefox dmgs

Need a small fix to libdmg first, to build with the old toolchain sysroot: https://github.com/mozilla/libdmg-hfsplus/pull/16

Pulls in https://github.com/mozilla/libdmg-hfsplus/pull/16 to handle the
older liblzma version in Debian 8 (used by the sysroot toolchain).

Pushed by bhearsum@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/93a8474201c9 Update libdmg-hfsplus to account for old liblzma. r=bhearsum

Not sure if it's automatically run thru Try somehow, but if not here's a run: https://treeherder.mozilla.org/jobs?revision=fd7741e5e96dcaec9d8d10809b7fef997963b24b&repo=try

https://hg.mozilla.org/mozilla-central/rev/93a8474201c9

Status: UNCONFIRMED → RESOLVED
Closed: 1 year ago
status-firefox135: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 135 Branch

:CosminS, that's just one commit of several that we'll need. I've reopened the bug.

I hope I've not been doing the process wrong, is there a one-changeset-per-bug rule? Should I be creating bugs to account for all the steps?

Status: RESOLVED → REOPENED
status-firefox135: fixed → ---
Ever confirmed: true
Resolution: FIXED → ---
Target Milestone: 135 Branch → ---

Generally best to limit the number of landing steps per bug yes, but if not we have the leave-open keyword.

Keywords: leave-open
Pushed by bhearsum@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/147811df7efd Add lzma to toolchain sysroot for libdmg-hfsplus r=bhearsum

Thanks for the tip :jcristau.

:bhearsum, any thoughts on whether we want this change just repackage, or for both build and repackage? (I'm honestly not even sure why we build a .dmg during the build instead of just leaving a tarball to be repackaged, it seems like a waste. But I'm not sure it's worth the effort to change.)

(In reply to Dave Vasilevsky [:vasi] from comment #33)

:bhearsum, any thoughts on whether we want this change just repackage, or for both build and repackage? (I'm honestly not even sure why we build a .dmg during the build instead of just leaving a tarball to be repackaged, it seems like a waste. But I'm not sure it's worth the effort to change.)

I think it's OK to ignore the build. It looks to me like it our signing task requires a dmg (or at the very least, it uses it). But we certainly don't ship it: the repackage task runs after that, and we ship the one coming out of it.

Dave, I've sorted out how to keep this on Nightly for the time being.

As a very quick overview, our repackage arguments are in one of our CI support files. The keys there are referenced by the top level CI configs. Eventually these are consumed by a script and fed through mach.

With all of this in mind you should be able to:

  • Add a new entry in PACKAGE_FORMATS that adds compression arguments
  • Adjust the two repackage kinds to use that for mozilla-central builds. You can use a by-release-type stanza similar to this one to do so.

The nightly release type will be used for autoland, mozilla-central, and try pushes based on those branches, so you should be able to verify this on try. (Other branches such as mozilla-beta and mozilla-release have their own types).

Thanks bhearsum, trying that now.

Working on it, but can't figure out how to test an optimized build. mach try fuzzy shows "repackage-macosx64/debug", but not "/opt" for some reason.

(In reply to Dave Vasilevsky [:vasi] from comment #38)

Working on it, but can't figure out how to test an optimized build. mach try fuzzy shows "repackage-macosx64/debug", but not "/opt" for some reason.

Ah, these jobs are not shown by default. You need ./mach try fuzzy --full. You'll want repackage-macosx64-shippable/opt.

  • Allows repackage_dmg to take a --compression argument (lzma, or
    default bzip2)
  • Enables lzma compression on nightlies

Testing

Release Note Request (optional, but appreciated)
[Why is this notable]: This bug will change the underlying compression in our macOS DMG packages.
[Affects Firefox for Android]: No
[Suggested wording]: Firefox Nightly macOS DMG packages now used LZMA for compression, shrinking download size and reducing installation time.
[Links (documentation, blog post, etc)]: n/a

Please note: this is explicitly being held on Nightly to bake for awhile. We do not yet know when we will have it ride to Beta.

relnote-firefox: --- → ?
Pushed by bhearsum@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/4a62c081066e Compress Mac nightly dmgs with lzma. r=bhearsum,taskgraph-reviewers

I've filed https://bugzilla.mozilla.org/show_bug.cgi?id=1935015 to track flipping this on for Beta & Release to keep things cleaner. I'm tentatively thinking that we'll do so in Firefox 136 (goes to Beta on February 3rd).

As for this bug, I think we can close it now. Thanks a ton Dave!!

Status: REOPENED → RESOLVED
Closed: 1 year ago → 1 year ago
Resolution: --- → FIXED
status-firefox135: --- → fixed
Target Milestone: --- → 135 Branch

Added to the Nightly relnotes. We can have the note ride whatever train bug 1935015 ends up riding out on when the time comes.

relnote-firefox: ? → nightly+
Blocks: 1935015

Riding the train to release in Fx136 with Bug 1935015

relnote-firefox: nightly+ → ---
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: