Open Bug 1494788 Opened 6 years ago Updated 2 years ago

add support for a tarball packaging format for mac

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Type:
enhancement

Tracking

(Not tracked)

People

(Reporter: mozilla, Unassigned)

References

Details

Currently, mac builds and l10n repacks create a dmg package. Mac build-signing then extracts the contents of that dmg into a .tar.gz and signs its contents; then the repackage task creates a dmg out of the tarball. The dmg->tar.gz step takes ~17 seconds, and happens once per locale, including en-US. that's ~1700 seconds of compute time, or ~28 minutes, per release or nightly graph. These run on the signing worker pool, which currently doesn't autoscale; we've been focusing on how to reduce mac signing as a bottleneck. This bug is one result of that focus. Two questions: 1. Do we need a specialized mac package from the build to include the icon, .ds_store, etc for the repackage task? I'm currently guessing yes. 2. If so, could we add support for a mac-specific tarball format? I imagine this would have the same layout as the dmg, but the package would be a tarball. I'm currently hoping that this is either faster than or equally fast as creating a dmg, but given the ~1700 second cost downstream, even a small increase in time could be an overall win. Once we resolve (2) we can adjust the task graph to use that new packaging format, and for the mac build-signing tasks to download+sign that file rather than the dmg.
I would bet good money that creating a tarball is way faster than creating a dmg, since we're using pretty janky tools to create dmgs on Linux. I don't see any reason why we couldn't just create a tarball containing the app bundle and extra bits. We'd probably have to tweak the post-signing packaging code to know how to invoke the dmg creation script with the right arguments. (I don't actually know what it does now, but presumably it has some way to properly rebuild the dmg with the icon and everything.)
Post-signing packaging code is `mach repackage`, I believe. It already takes the tarball from the signing task: https://taskcluster-artifacts.net/dzPbg3u2QOGy9RqBmHCkXQ/0/public/logs/live_backing.log
Keywords: in-triage
See Also: → 922241
Keywords: in-triage
With :kmoir's consent, I'm reviewing the backlog for bugs in FFx Build System to see if we can do some things to clean up the backlog. This bug appears to be an enhancement or feature request. If so, please set the bug's severity to `enhancement` and/or add the `feature` keyword depending on if this is an improvement to an existing thing, or a new feature/tool/system.
Flags: needinfo?(aki)
Severity: normal → enhancement
Flags: needinfo?(aki)

ni mshal and chmanchester so can provide any feedback on this approach in comment 1

Flags: needinfo?(mshal)
Flags: needinfo?(cmanchester)

Does the repackage task have a source checkout? What about a configure run? The dsstore/background.png/icons all come from the source tree, so they don't necessarily need to be passed in with the tarball from the build task. However we would need to be able to compute MOZ_BRANDING_DIRECTORY in the repackage task, which currently comes from configure. Those files are just copied into specific locations in the tarball though. It's possible they could just be converted to an install manifest and copied like other build files. It would be nice to be able to remove the OSX-specific packing bits still in Makefiles, but I don't think that's a requirement to implement this. In theory we can just change the MOZ_PKG_FORMAT to BZ2 or TGZ and see what breaks.

The only other consideration I can think of is that some clients like mozregression currently expect .dmg files, so if the repackaging doesn't happen on every build (like in autoland), then those clients would need to be updated to look for .tar.gz as well.

Flags: needinfo?(mshal)

(In reply to Michael Shal [:mshal] from comment #5)

Does the repackage task have a source checkout? What about a configure run?

I think so. This central repackage log says yes.

The dsstore/background.png/icons all come from the source tree, so they don't necessarily need to be passed in with the tarball from the build task. However we would need to be able to compute MOZ_BRANDING_DIRECTORY in the repackage task, which currently comes from configure. Those files are just copied into specific locations in the tarball though. It's possible they could just be converted to an install manifest and copied like other build files. It would be nice to be able to remove the OSX-specific packing bits still in Makefiles, but I don't think that's a requirement to implement this. In theory we can just change the MOZ_PKG_FORMAT to BZ2 or TGZ and see what breaks.

Catlee has a patch to create a tar.bz2 that I'm testing in https://github.com/escapewindow/gecko/pull/1/commits/831ee472e80d04f0071c621a9cbc9c84e0e031b9 .

The only other consideration I can think of is that some clients like mozregression currently expect .dmg files, so if the repackaging doesn't happen on every build (like in autoland), then those clients would need to be updated to look for .tar.gz as well.

+1

It looks like a repackaged DMG from a tar.bz2 looks different (no firefox icon on the mounted dmg, tiny icons inside the dmg) https://queue.taskcluster.net/v1/task/CcwEM-MrTRSuvKNSQkKs9A/runs/0/artifacts/public/build/target.dmg

(In reply to Aki Sasaki [:aki] from comment #7)

It looks like a repackaged DMG from a tar.bz2 looks different (no firefox icon on the mounted dmg, tiny icons inside the dmg) https://queue.taskcluster.net/v1/task/CcwEM-MrTRSuvKNSQkKs9A/runs/0/artifacts/public/build/target.dmg

The link to the patches in #c6 doesn't work anymore, but in the build log or repackage log I don't see anything that tries to install the dsstore/icons/background.png. Where should that be happening now?

In addition to mozregression we have artifact builds that currently consume dmgs but we'd actually prefer tar.bz2 there as we currently spend significant time mounting and process the contents of dmgs during artifact builds. In other words, we'd like to see this happen but there would be a little work on the build system side that would have to happen around the same time.

We should also keep in mind developers pushing to try and testing those builds locally, we would want to make this change as transparent as possible for that use case.

Flags: needinfo?(cmanchester)

(In reply to Michael Shal [:mshal] from comment #8)

(In reply to Aki Sasaki [:aki] from comment #7)

It looks like a repackaged DMG from a tar.bz2 looks different (no firefox icon on the mounted dmg, tiny icons inside the dmg) https://queue.taskcluster.net/v1/task/CcwEM-MrTRSuvKNSQkKs9A/runs/0/artifacts/public/build/target.dmg

The link to the patches in #c6 doesn't work anymore, but in the build log or repackage log I don't see anything that tries to install the dsstore/icons/background.png. Where should that be happening now?

Ah, probably rebased out of existence. Should be https://github.com/escapewindow/gecko/pull/1/commits/42ba8724599c2dab0e22cd31dbd677aabf8d9157 , or the commit in https://github.com/escapewindow/gecko/pull/1/ that starts with catlee.

I'm guessing we may have to move that dsstore/icons/background.png logic into the repackage logic in-tree.

My attempt to make make_dmg.py able to create either a dmg or a dmg-formatted tarball: https://hg.mozilla.org/try/rev/c23f355d21926226ac5a0f743c3260fe17c24000

(In reply to Aki Sasaki [:aki] from comment #11)

My attempt to make make_dmg.py able to create either a dmg or a dmg-formatted tarball: https://hg.mozilla.org/try/rev/c23f355d21926226ac5a0f743c3260fe17c24000

Try push here. Looking at it, we're probably still going to make a staged.hfs on linux, which is likely not what we want.

Severity: normal → S3
You need to log in before you can comment on or make changes to this bug.