1046306 - [tracking bug] make necessary changes to OS X builds/signing to support 10.9.5

Status: RESOLVED WORKSFORME

(Release Engineering :: General, defect)

Description

[[e-mail] (:bhearsum)]

Attachment: Technical Note TN2206: OS X Code Signing In Depth.pdf

***********Please note that these changes are under NDA, so we cannot talk about this in public at this time.***********

I got an e-mail from Apple today about some changes coming in 10.9.5. The tl;dr appears to be that we need to use v2 signatures (requires 10.8.5 or higher), and some fairly significanty changes to how we arrange our builds. Full text of the mail is below, as well as the attachment that they reference.

I'm in the process of getting a 10.8 and 10.9 machine set-up to test signing. There's no preview builds of 10.9.5 available yet, but I'm hoping that simply by signing with a v2 signature we can make some progress.

I'm trying to get access to a 10.9.5 image, as well as a release date for it so we know what kind of timeline we're working on. Based on release history (http://en.wikipedia.org/wiki/OS_X_Mavericks#Releases) I'm guessing they're shipping in August or September - but that's just my own speculation.

Here's the text of the e-mail:
This information is covered under your developer program NDA.

Team Mozilla:

Beginning with OS X version 10.9.5, there will be changes in how OS X recognizes signed apps. Signatures created with OS X version 10.8.5 or earlier ("v1 signatures") are obsoleted and will no longer be recognized by
+Gatekeeper. To ensure your apps will run on updated versions of OS X they must be signed on OS X version 10.9 or later ("v2 signatures").
+
If your team is using an older version of OS X to build your code, sign your app using OS X version 10.9 or later using the codesign tool to create v2 signatures. Apps signed with v2 signatures will work on older versions of OS
+X.
+
Your bundle will need to be structured according to the signature evaluation requirements for OS X version 10.9 or later. Some considerations include:
- Signed code should only be placed in directories where the system expects to find signed code
- Resources should not be located in directories where the system expects to find signed code
- The -resource-rules flag and ResourceRules.plist are not supported.

We've noticed that you use custom resource rules in your application. If you feel its important for Apple to understand why custom resource rules are used in your build process, please file a bug at https://bugreport.apple.com
+and add the phrase ‘Resource Rules’ in the title.
+
To ensure your current and upcoming releases work properly with Gatekeeper, please test on OS X version 10.10 (Seed 5 or later) and OS X version 10.9.5.

For more information see  OS X Code Signing In Depth (draft attached)

Comment 1

[[e-mail] (:bhearsum)]

There's a bunch of questions we need to answer. Off the top of my head:
* Does this affect us since we don't distribute through the App Store?
* Which versions (including minor versions) of OS X support v2 signatures? This could have implications for our ability to support older versions such as 10.6 and 10.7.
** What happens if the version of OS X running doesn't support v2 signatures? Is the build runnable? Does it throw a warning? We need answer these both for new installs and updates IMO.
* Do we need to totally change the way we package, per the attached document?
* Which version of OS X *should* we sign on? We settled on 10.6 last time because signatures done there were valid on 10.6 and up. We should test both 10.8 and 10.9 to see which gives us the best compatibility.

Comment 2

[:Gavin Sharp [email: [e-mail]]]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> * Does this affect us since we don't distribute through the App Store?

Yes - we need to be able to run with Gatekeeper enabled regardless.

CCing some folks from bug 723176.

Comment 3

[Stephen A Pohl [:spohl]]

(In reply to Ben Hearsum [:bhearsum] from comment #1)
> * Which versions (including minor versions) of OS X support v2 signatures?

Apple's original email states that:

> Apps signed with v2 signatures will work on older versions of OS X.

Hopefully, this goes back to at least 10.6. We'll need to verify, of course.

Comment 4

[[e-mail] (:bhearsum)]

Anyone who's helping here: I can get you a Mac Development certificate that should allow you to do whatever testing you need locally. Just let me know.

Comment 5

[[e-mail] (:bhearsum)]

Steven just pointed out to me offline that I misread the mail re: which versions we can sign on. Turns out we must sign on 10.9. That makes one thing simpler, I guess.

Comment 6

[Steven Michaud [:smichaud] (Retired)]

For what it's worth, I just tried to sign a custom build on OS X 10.9.4, and it didn't work at all.

Here's the command I used:

codesign -s "Steven Michaud" Nightly.app

Here's the error I got:

Nightly.app: code object is not signed at all
In subcomponent: /Volumes/Nightly/Nightly.app/Contents/MacOS/application.ini

(I'd previously played the tricks you need with hdiutil to mount the build readwrite.  I checked that it was writable, so that's not the problem.  Exactly the same thing works just fine on OS X 10.7.5.)

Comment 7

[Steven Michaud [:smichaud] (Retired)]

> Exactly the same thing works just fine on OS X 10.7.5.

And on OS X 10.8.5.

Comment 8

[(not currently active) Ted Mielczarek]

(In reply to Steven Michaud from comment #6)
> Nightly.app: code object is not signed at all
> In subcomponent: /Volumes/Nightly/Nightly.app/Contents/MacOS/application.ini

I wonder if this falls under: "- Resources should not be located in directories where the system expects to find signed code"? This seems like a pretty significant change, we'd have to move all of our non-code resources to a separate directory under the bundle.

Comment 9

[(not currently active) Ted Mielczarek]

Yeah, wow, this is going to be a pain in the ass. From the section of the PDF here entitled "Code signing changes in OS X Mavericks", under Table 3: "These places are expected to contain only code. Putting arbitrary data files there will cause them to be rejected (since they're unsigned)."

Comment 10

[[e-mail] (:bhearsum)]

I just created #macstuff to talk about this in, after we accidentally started talking in a public channel.

Comment 11

[[e-mail] (:bhearsum)]

I just spoke briefly with Michael from Apple. He does not have a preview build for us, nor an exact release date. I asked him if we were talking about "weeks" or "months", and he said "weeks".

Comment 12

[Steven Michaud [:smichaud] (Retired)]

Here's a chart (from a Wikipedia article) of when the Mavericks minor versions were released:
http://en.wikipedia.org/wiki/OS_X_Mavericks#Releases

10.9.4 was released a little more than a month after 10.9.3, but the other releases were farther apart.  I frankly don't think we can trust anything Apple says about when they'll do something.  But their past behavior is usually a pretty good guide.

But though Apple surely has its own plans (which they'll never tell us about), there *is* one circumstance that might force them to change them (and release 10.9.5 "early") -- a big, public security hole that they need to fix quickly.

Comment 13

[Steven Michaud [:smichaud] (Retired)]

Another thing:

Apple has always made available several developer seeds (at the Mac Member Center) for each new minor version before its release.  So far none have been made available for OS X 10.9.5.  (I checked again just now.)

Comment 14

[[e-mail] (:bhearsum)]

Madhava, Chad - we talked about this after the channel meeting today, Lawrence wanted to make sure you're both aware of it.

Comment 15

[[e-mail] (:bhearsum)]

Another thing we need to watch out for is the ability to exclude files from the signature. You can see the list of things we currently exclude in https://mxr.mozilla.org/mozilla-central/source/browser/app/macbuild/Contents/_CodeSignature/CodeResources

Of particular note:
defaults/ - which contains prefs/channel-prefs.js. If we can't exclude this file, I don't think we can do cross-channel updates anymore (such as shipping RCs to beta users).
distribution/ - this needs to be excluded from the signature in order for partner builds to be able to update.

On IRC, spohl mentioned that he thinks these files can live under Resources and be excluded, so hopefully that will work out.

Connor, I'm cc'ing you as a heads up because of the potential impact to partner builds.

Comment 16

[Stephen A Pohl [:spohl]]

(In reply to Ben Hearsum [:bhearsum] from comment #15)
> Of particular note:
> defaults/ - which contains prefs/channel-prefs.js. If we can't exclude this
> file, I don't think we can do cross-channel updates anymore (such as
> shipping RCs to beta users).

Tracked in bug 1047728.

> distribution/ - this needs to be excluded from the signature in order for
> partner builds to be able to update.

Tracked in bug 1047738, bug 1047740 and bug 1047742.

> On IRC, spohl mentioned that he thinks these files can live under Resources
> and be excluded, so hopefully that will work out.

That was my hope yesterday. Today, we've pretty much confirmed that Apple will no longer support omitting any files from the top-level .app signature using v2 signatures. I'm afraid we might no longer be able to modify any files inside the .app without breaking the signature.

Comment 17

[Stephen A Pohl [:spohl]]

(In reply to Stephen Pohl [:spohl] from comment #16)
> (In reply to Ben Hearsum [:bhearsum] from comment #15)
> > On IRC, spohl mentioned that he thinks these files can live under Resources
> > and be excluded, so hopefully that will work out.
> 
> That was my hope yesterday. Today, we've pretty much confirmed that Apple
> will no longer support omitting any files from the top-level .app signature
> using v2 signatures. I'm afraid we might no longer be able to modify any
> files inside the .app without breaking the signature.

The 'precomplete' file in the root of the Firefox.app bundle kept intriguing me, so I ran a quick test and copied a folder into the root of the .app bundle after it was signed and... it works! The signature is still valid! This might solve our problems regarding distribution partners, channel-prefs.js etc. since we can modify the files there after the bundle was signed. I will update the appropriate bugs and ask the right people for more info to confirm.

Comment 18

[:Gavin Sharp [email: [e-mail]]]

If 10.9.5 will be released in "weeks", we're likely going to need to target any in-product changes to Firefox 32 (releasing Sept 2), which might be quite aggressive.

Robert, do you have a good sense of the feasibility of that timeline for the product changes?

Ben, what are you thoughts regarding targeting 32 from a build infra perspective (i.e. other dependencies here)?

Comment 19

[Robert Strong (they/them - no direct email)]

Gavin, my sense of the feasibility is that this will be extremely difficult to implement these changes in that timeline. I've looped bsmedberg in (he is back from pto on Monday), Stephen looped Josh Aas in, lmandel has sent an email to Apple (I'll cc you on the thread that I was added to on Friday), I gave Johnath a heads up on Friday, and as shown by comment #17 Stephen is investigating possible solutions to the problems we've identified.

Now that much of what will need to be done has been identified I think we should get several people identified as resources that can take on these tasks (keeping in mind that this is still under NDA) and meet to discuss what will need to be done (I will be out most of Monday afternoon) as well as assign the tasks.

Comment 20

[[e-mail] (:bhearsum)]

(In reply to :Gavin Sharp [email: [e-mail]] from comment #18)
> If 10.9.5 will be released in "weeks", we're likely going to need to target
> any in-product changes to Firefox 32 (releasing Sept 2), which might be
> quite aggressive.
> 
> Robert, do you have a good sense of the feasibility of that timeline for the
> product changes?
> 
> Ben, what are you thoughts regarding targeting 32 from a build infra
> perspective (i.e. other dependencies here)?

At this time, I think the only infra change we need is to move the signing servers to 10.9. If that holds true, I think we'll be well clear. I've got an initial patch for the signing changes, but it needs more testing and review. Once landed, I expect it to take a day or two to move everything over. I'm out this Monday and Friday, but if things go well we should have all of this done by the 13th.

Comment 21

[Stephen A Pohl [:spohl]]

lmandel confirmed on IRC that Apple has made this info public now.

Comment 22

[Mike Hommey [:glandium]]

It seems to me part of bug 1047728, bug 1047738, bug 1047740 and bug 1047742 would just be to change what gecko considers as the GREdir.

Comment 23

[Stephen A Pohl [:spohl]]

(In reply to Mike Hommey [:glandium] from comment #22)
> It seems to me part of bug 1047728, bug 1047738, bug 1047740 and bug 1047742
> would just be to change what gecko considers as the GREdir.

Unfortunately, it's a bit trickier than this. We already have to point the GREdir at <bundle>/Contents/Resources to make Firefox launch and run in bug 1050944, so we can't reuse the GREdir for <bundle>/MozResources for the bugs that you mention...

Comment 24

[Mike Hommey [:glandium]]

Why not put everything in the same directory like now?

Comment 25

[Mike Hommey [:glandium]]

(besides binaries)

Comment 26

[Stephen A Pohl [:spohl]]

bsmedberg brought up the same question and I don't really have a good answer. Basically, what put us in this position in the first place was that we didn't have a standard Apple bundle structure. By putting everything into yet another non-standard directory might just cause the same problems again in the future. For a moment when we first heard about v2 signatures, we thought that this was the end for any files that could change within a bundle. We eventually found a workaround by putting files next to Contents in the root of the bundle (i.e. MozResources), but we have no guarantee that Apple won't close this loophole. If they do, it will at least not affect the rest of Firefox again.

My initial instinct was: Since we already have to change a whole bunch of stuff, we might as well make our bundle more Apple-compliant and (hopefully) avoid further problems in the future. But if there are strong voices in favor of putting everything in MozResources, we can certainly explore this idea.

Comment 29

[Robert Strong (they/them - no direct email)]

All depends on bugs resolved and all patches landed on aurora in the Mac V2 signing combined patch in bug 1047584 so resolving wfm

Comment 31

[kbotelho]

It appears that this is still unresolved for the "plugin-container.app" within the bundle, if you go to the following link you will see that that there is still a signing issue with "plugin-container.app", as it consistently prompts to allow incoming connections for the OS firewall.

https://bluejeans.com/10305/3230/browser

Comment 32

[Stephen A Pohl [:spohl]]

(In reply to kbotelho from comment #31)
> It appears that this is still unresolved for the "plugin-container.app"
> within the bundle, if you go to the following link you will see that that
> there is still a signing issue with "plugin-container.app", as it
> consistently prompts to allow incoming connections for the OS firewall.
> 
> https://bluejeans.com/10305/3230/browser

Please file a new bug (blocking this one) with detailed instructions how to reproduce the problem and CC me. It appears that this site requires an account and a plugin to be installed. However, the plugin doesn't seem to install on OSX 10.10. Aside from detailed steps to reproduce, please also include your OSX version and Firefox version in your bug report. Thanks!