Closed Bug 1351474 Opened 7 years ago Closed 7 years ago

Downgrade dmg tools to avoid corrupt dmgs in cross-osx builds

Categories

(Firefox Build System :: General, enhancement)

Product:
Firefox Build System
Component:
General
Type:
enhancement
Priority:
Not set
Severity:
normal

Tracking

(firefox55 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla55
Tracking Flags:
Tracking Status
firefox55 --- fixed

People

(Reporter: mshal, Assigned: mshal)

References

Details

Attachments

(2 files)

Bug 1351474 - use Mozilla fork of libdmg-hfsplus;
59 bytes, text/x-review-board-request
ted
: review+
Details
Bug 1351474 - Downgrade dmg tools to avoid corrupt dmgs;
59 bytes, text/x-review-board-request
ted
: review+
Details 
Bug 1337393 upgraded the dmg tools, but unfortunately that produces dmg files that don't fsck properly:

$ fsck_hfs  /dev/disk1s3
** /dev/rdisk1s3 (NO WRITE)
   Executing fsck_hfs (version hfs-305.10.1).
** Checking non-journaled HFS Plus Volume.
   The volume name is Nightly
** Checking extents overflow file.
   Unused node is not erased (node = 108)
** Checking catalog file.
   Unused node is not erased (node = 108)
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
   Unused node is not erased (node = 22)
** Checking volume bitmap.
** Checking volume information.
** The volume Nightly was found corrupt and needs to be repaired.

That bug primarily was needed to preserve permissions on files. We can go back to the version of dmg that isn't corrupted and instead backport the patches needed to preserve permissions.
Blocks: 1347579
This seems to work, though I don't know if managing patches makes sense vs. just publishing our own yet-another-fork of libdmg and using that.

$ fsck_hfs /dev/disk1s3
** /dev/rdisk1s3 (NO WRITE)
   Executing fsck_hfs (version hfs-305.10.1).
** Checking non-journaled HFS Plus Volume.
   The volume name is Nightly
** Checking extents overflow file.
** Checking catalog file.
** Checking multi-linked files.
** Checking catalog hierarchy.
** Checking extended attributes file.
** Checking volume bitmap.
** Checking volume information.
** The volume Nightly appears to be OK.
(In reply to Michael Shal [:mshal] from comment #3)
> This seems to work, though I don't know if managing patches makes sense vs.
> just publishing our own yet-another-fork of libdmg and using that.

I think we should publish our own fork and create a branch with the patches we need committed. That will at least make it easier for others to build binaries from the same code we're using.
Comment on attachment 8852273 [details]
Bug 1351474 - use Mozilla fork of libdmg-hfsplus;

https://reviewboard.mozilla.org/r/124494/#review127208

Can you just fork libdmg-hfsplus and commit these patches on a `mozilla` branch, then point the script at that?
Attachment #8852273 - Flags: review?(ted) → review-
Comment on attachment 8852274 [details]
Bug 1351474 - Downgrade dmg tools to avoid corrupt dmgs;

https://reviewboard.mozilla.org/r/124496/#review127210

Feel free to carry this r+ over if you build new binaries and update the manifest.
Attachment #8852274 - Flags: review?(ted) → review+
:gerv, is libdmg-hfsplus something we can import into Mozilla's github repo? It's GPL-v3: https://github.com/andreas56/libdmg-hfsplus/blob/from_zarvox/LICENSE

We don't distribute it with the release, but it would be used to build the .dmg file that we release for OSX. When I asked github-owners@mozilla about importing it into the github.com/mozilla umbrella, rsoderberg was concerned that the license would be an issue.
Flags: needinfo?(gerv)
To clarify a point: do we *already* use this, and we're just talking about forking it so we can local-patch it to be more useful to us? Or, are we considering using this for the first time?
Note by "Mozilla's github repo", I mean this organization: https://github.com/mozilla

Not the gecko-dev github repository.
(In reply to Richard Soderberg [:atoll] from comment #8)
> To clarify a point: do we *already* use this, and we're just talking about
> forking it so we can local-patch it to be more useful to us? Or, are we
> considering using this for the first time?

We do not currently use it - right now we use OSX's native tools to build a .dmg file. We are interested in distributing the .dmg from a Linux cross-compiled build, which would use libdmg-hfsplus to build the final .dmg since Linux does not have built-in tools to handle that. We have to patch it at the moment due to various issues with other versions we've tried.
Just as a test, I tried making a .ISO using normal tools and renaming it to .DMG. It worked fine for me, because the HDI toolkit in OS X lets the internal filetype override the extension.

I assume by this point Firefox is already code-signed, so you don't need the 'DMG' file to be codesigned, and so making a .ISO and renaming it should be fine enough *in theory*. Obviously I didn't do heavy testing, but if it provides you a sidestep for this entire problem, win is you.
There aren't any licensing concerns here. We're just using it as a build tool. GCC is also GPL-licensed.
Sure, that's fine - I only ask because we have zero active GPLv3 repositories in the Mozilla official Github org, and I'm not sure if that's by enforcement or by accident.
(In reply to Richard Soderberg [:atoll] from comment #13)
> Sure, that's fine - I only ask because we have zero active GPLv3
> repositories in the Mozilla official Github org, and I'm not sure if that's
> by enforcement or by accident.

Pure speculation, but our license policy requires that Mozilla projects are either MPL 2 or Apache 2:
https://www.mozilla.org/en-US/MPL/license-policy/

So it would be against Mozilla policy to start a *Mozilla* project using the GPL. We use plenty of other GPL software for various things, and there shouldn't be any issues with us publishing a fork of existing GPL software that we use with the patches we need.
(In reply to Michael Shal [:mshal] from comment #7)
> :gerv, is libdmg-hfsplus something we can import into Mozilla's github repo?
> It's GPL-v3:
> https://github.com/andreas56/libdmg-hfsplus/blob/from_zarvox/LICENSE
> 
> We don't distribute it with the release, but it would be used to build the
> .dmg file that we release for OSX. When I asked github-owners@mozilla about
> importing it into the github.com/mozilla umbrella, rsoderberg was concerned
> that the license would be an issue.

Code that doesn't ship with a product just needs to be under an open source license, so GPLv3 is fine. (See https://www.mozilla.org/media/img/mozorg/mpl/license-policy-flowchart.3d01662c2c7d.png - you end up at "use the license it already uses".) And of course we need to make sure that we aren't using it in violation of the license, which I presume we are not.

Gerv
Flags: needinfo?(gerv)
Sounds like we are ok to move this over then. I think we'll just want the Releng group to have access. (I would suggest the build config team as well, but it doesn't look like we have a group for that in github yet.) rsoderberg, do you need any other info?
Flags: needinfo?(rsoderberg)
Just a follow up on the GitHub-owners thread summarizing what we learned, so everyone's aware.
Flags: needinfo?(rsoderberg)
Comment on attachment 8852273 [details]
Bug 1351474 - use Mozilla fork of libdmg-hfsplus;

https://reviewboard.mozilla.org/r/124494/#review128218

Thanks!
Attachment #8852273 - Flags: review?(ted) → review+
Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/d6ec05994609
use Mozilla fork of libdmg-hfsplus; r=ted
https://hg.mozilla.org/integration/autoland/rev/0da41fa5f8bc
Downgrade dmg tools to avoid corrupt dmgs; r=ted
https://hg.mozilla.org/mozilla-central/rev/d6ec05994609
https://hg.mozilla.org/mozilla-central/rev/0da41fa5f8bc
Status: NEW → RESOLVED
Closed: 7 years ago
status-firefox55: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla55
Depends on: 1356993
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: