Closed Bug 752613 Opened 13 years ago Closed 13 years ago

Sign mac Firefox bundles with Developer ID

Categories

(Release Engineering :: General, defect)

Product:
Release Engineering
Component:
General
Platform:
All
macOS
Type:
defect
Priority:
Not set
Severity:
normal

Tracking

(Not tracked)

Status:
RESOLVED DUPLICATE of bug 730924

People

(Reporter: benjamin, Assigned: bhearsum)

References

(Blocks 1 open bug)

Details

I was contacted by somebody at Apple and they are very interested in getting the Firefox packages signed using Apple Developer ID:

https://developer.apple.com/resources/developer-id/

"Mozilla is the only browser we know of that isn't in the process of being signed, so we'd love to have you on board as soon as possible."

Asa, should I write up a feature page for this, or is the bug sufficient?
OS: Windows 7 → Mac OS X
Hardware: x86_64 → All
We're already signing builds with a converted key purchased from Thawte. Why do we have to use an Apple one? When we looked at this a few months ago we didn't see any benefit at the time.
With the upcoming 10.8 and it's Gatekeeper, only Apple Developer IDs are really trusted. 
I haven't checked it yet, but i'm assuming that it will come with a restrictive default.
Because by default the new version of Mac OS X will only allow developer-id-signed software onto the machine, if I understand it correctly. See the screenshot at 

http://www.apple.com/macosx/mountain-lion/security.html

Where users can choose to "Allow applications downloaded from:

* Mac App Store
* Mac App Store and identified developers
* Anywhere"

If our bundles are not signed with Developer ID we will only install in the "anwhere" bucket.
I don't know what the *default* setting will be.
(In reply to comment #2)

> We're already signing builds with a converted key purchased from Thawte.

We *might* be able to do this, but I've already found some evidence that it won't work.  I'll open a new bug on this subject in the next day or two.

Have you done any testing on Mountain Lion?  It needs to have been on DP3 or later -- DP3 was the first build that complained when you tried to run an unsigned copy of FF.  And you need to test with FF 11 or older -- FF 12 runs without being signed (which I think is an Apple bug).
In any case I think we should initiate the process of getting an Apple ID from Apple as soon as possible, even if we might not need it.

There may be delays, and other complications.
(In reply to Steven Michaud from comment #5)
> (In reply to comment #2)
> 
> > We're already signing builds with a converted key purchased from Thawte.
> 
> We *might* be able to do this, but I've already found some evidence that it
> won't work.  I'll open a new bug on this subject in the next day or two.
> 
> Have you done any testing on Mountain Lion?  It needs to have been on DP3 or
> later -- DP3 was the first build that complained when you tried to run an
> unsigned copy of FF.  And you need to test with FF 11 or older -- FF 12 runs
> without being signed (which I think is an Apple bug).

Marcia tested a signed build on 10.8 a couple of weeks ago - I don't know which DP that was. I can give you a build signed with a Thawte-purchased certificate if you'd like to do some additional testing.
(In reply to comment #3 and comment #4)

Right now the default (on Mountain Lion) is:

* Mac App Store and identified developers
> Marcia tested a signed build on 10.8 a couple of weeks ago

DP3 has only been out a few weeks.  That might not have been the version Marcia tested with.

> I can give you a build signed with a Thawte-purchased certificate if
> you'd like to do some additional testing.

Please give me a FF 11 build signed with this cert.  We can figure out offline how you can get it to me.
I've now tested with Ben Hearsum's signed copy of an FF
mozilla-central nightly (signed with the Thawte signing cert).
Mountain Lion DP3 doesn't recognize the signature -- you get the
"unrecognized developer" error message.  This is exactly the same
result as you get with an unsigned copy of the same mozilla-central
nightly.

So we'll probably need an Apple ID from Apple, and should start the
process of getting one right away.
For those testing app-signing on Mountain Lion:

I've been telling people that FF 12 runs unsigned on Mountain Lion.
Well *my* copy does, but it turns out not to be true in general.

Here's the real problem (which is definitely Apple's bug):

If you download an FF dmg over an http connection, the browser (FF or
other browsers) will tag it with one or more extended attributes to
indicate that it's been downloaded from the web.  If you mount this
dmg and drag the FF distro from it to your hard drive, the first time
you run it (on FF Lion and below) you'll be warned that it was
dowloaded from the web on such-and-such a date, and asked if you want
to trust it.  

You see the same dialog on Mountain Lion (DP3 and above) if the app is
signed with an Apple ID from Apple.  But if it's unsigned (or has been
signed with Ben's Thawte cert or my test signing cert), you get the
"unrecognized developer" warning.

If you download an FF dmg over an sftp/ftp connection, the destination
copy doesn't get any extended attributes.  If you mount this dmg and
drag the FF distro from it to your hard drive, you won't see any
warnings the first time you run it.

*And you won't see any warnings on Mountain Lion, even if the app is
unsigned.*

I'll open a bug about this in the next few days.
Benjamin, thanks for filing this. Steven and I have been chatting about this for a week. We need to move on this ASAP. We don't need a feature page. This bug will do. 

bhearsum, are you the right person to get the cert from Apple? Benjamin, can you help Ben with your contact?
I am absolutely throttled with things right now. I will have time to get up to speed on this late this week or next week. If this needs to start moving before then, talk with Chris AtLee or John O'Duinn.
(Following up comment #11)

> If you download an FF dmg over an http connection, the browser (FF
> or other browsers) will tag it with one or more extended attributes
> to indicate that it's been downloaded from the web. ...

> If you download an FF dmg over an sftp/ftp connection, the
> destination copy doesn't get any extended attributes. ...

This is slightly wrong:

You get the extended attributes if you download a dmg *in a browser*
(whether over an http connection or an ftp one).

You don't get them if you download the dmg using a simple command-line
tool like sftp or ftp (or perhaps a very simple "browser").
It is possible that Mountain Lion will become available at WWDC 2012 which starts in less than 5 weeks. We need to have builds signed with an Apple Developer ID tested and working before then.
By the way, just so that people know:

The infrastructure for signing builds already exists -- it was a project done by Erick Dransch (one of our interns).  As far as I know any signing cert can be used with it.

What we (urgently) need is a signing cert from Apple.
cc'ing John O'Duinn and Chris Atlee
Steven: I guess I have been updating all of my builds on 10.8 so I did not see this with the latest preview build.

(In reply to Steven Michaud from comment #11)
[snip]
(In reply to comment #18)

Marcia, there's another factor that might have influenced your tests results.

Do you have a /var/db/DetachedSignatures file?  If so then you'll need to follow the procedure I'll describe in my next comment.

One way this file can be created is if you work around the "unrecognized developer" by right-clicking on an FF distro and choosing "open".  Do this once, and subsequently you'll never see the "unrecognized developer" warning for any app that has the same bundle identifier.

You also (of course) need to make sure you have the default setting for "Allow applications downloaded from", which is:

* Mac App Store and identified developers
Here's a list of the things you need to know/do in order to run
meaningful tests of app signing on MountainLion (DP3 and above).  I'll
include the things I've already talked about in comment #11 and
comment #14.

For those testing app-signing on Mountain Lion:

1) Be sure to test with apps dragged out of dmg images that were
   downloaded using a web browser.

   Dmg images downloaded using an sftp/ftp client (command-line or
   otherwise) won't "work":  Thanks to an Apple bug, apps dragged out
   of these images always run, even if unsigned.

2) If you have a /var/db/DetachedSignatures file, perform the
   following steps before running any tests.  (I'm not sure if all
   these steps are necessary.  I haven't yet had time to do sufficient
   testing to find out.)

   a) sudo rm /var/db/DetachedSignatures

   b) Delete all copies of FF (or whatever app you're going to be
      testing) that exist in "standard" locations (e.g. the desktop or
      the Applications folder) and empty the trash.

   c) Rebuild the launch services database by running the following
      command (without line breaks):

/System/Library/Frameworks/CoreServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain user -domain local -domain system

   d) Restart your computer.
Component: Release Engineering → Release Engineering: Automation (General)
QA Contact: release → catlee
Here are 3 copies of the FF 12 distro that I signed using Apple's codesign utility and my codesigning cert from Apple.  One I signed on OS X 10.6.8, another I signed on OS X 10.7.3, and the third I signed on OS X 10.8 DP3.

https://people.mozilla.com/~stmichaud/SigningTests/

All three run fine on OS X 10.8 (DP3), and also on OS X 10.5.8.

Please try them.

More in my next comment about how I made these builds.
Here's how I made the builds from comment #21:

1) hdiutil convert "Firefox 12.0.dmg" -format UDRW -o stage1.dmg
2) hdiutil resize -size 100m stage1.dmg
3) hdiutil attach stage1.dmg -readwrite
4) codesign -s "Steven Michaud" /Volumes/Firefox/Firefox.app	
5) hdiutil detach /dev/disk1
6) hdiutil convert stage1.dmg -format UDBZ -o "Firefox 12.0-signed.dmg"
Assignee: nobody → bhearsum
Our current setup for signing windows binaries uses 3 distinct sets of keys:
- try/depend keys: for builds that happen on try server or for incremental/depend builds on any branch
- nightly keys: for nightly builds on certain branches like mozilla-central and aurora
- release keys: for beta and release builds

We should mimic this setup for our OSX signing. We can share the same self-signed keys for try/depend since those aren't intended to be valid anyway.

Are we able to get different keys from apple for our nightly and release builds?
> Are we able to get different keys from apple for our nightly and release builds?

I expect so, but I don't really know.  You might need to get them for different Mozilla identities -- e.g. "Mozilla", "Mozilla Nightlies", "Mozilla Try/Depend".

You (or someone) should contact Apple right away, to start the process of getting at least one Apple ID for "Mozilla".  Alex might be able to suggest who (or which department) at Apple you should contact.
(Following up comment #21 and comment #22)

> https://people.mozilla.com/~stmichaud/SigningTests/

I've now added signed distros of today's mozilla-central nightly here -- one signed on OS X 10.7.3, the other signed on OS X 10.8 DP3.  Both work find on OS X 10.8 (DP3) and OS X 10.5.8.

Signing failed on OS X 10.6.8 with the following errors:

codesign_allocate: for architecture x86_64 object: /Volumes/Nightly/FirefoxNightly.app/Contents/MacOS/firefox malformed object (unknown load command 9)
/Volumes/Nightly/FirefoxNightly.app: object file format invalid or unsuitable

I imagine this happens because we now build mozilla-central nightlies on OS X 10.7. Our build process probably adds a segment to the firefox binary that is unrecognized on OS X 10.6.8.

There's probably some way to manipulate the build process to fix this ... but I'm not sure we really care.
(In reply to Steven Michaud from comment #24)
> > Are we able to get different keys from apple for our nightly and release builds?
> 
> I expect so, but I don't really know.  You might need to get them for
> different Mozilla identities -- e.g. "Mozilla", "Mozilla Nightlies",
> "Mozilla Try/Depend".
> 
> You (or someone) should contact Apple right away, to start the process of
> getting at least one Apple ID for "Mozilla".  Alex might be able to suggest
> who (or which department) at Apple you should contact.

Is there any reason to believe we don't just pay the $100 to get a Mac Developer license at https://developer.apple.com/programs/mac/ and then use Xcode to generate a cert per the link in Comment 0?

Who can make that purchase on behalf of Mozilla so that we can move forward with verifying our understanding of code signing in preparation for Mountain Lion? Chris?
Just to set expectations, this is a top priority and we need be ready to get Firefox signed in preparation for Mountain Lion, either in FF13 or a FF13.0.1.
I am on buildduty this week and won't be able to touch this until either Friday or Monday. It will be my primary at that point, and there's still plenty of time to deal with this. We know how to sign builds already, the only questions are which certificate to sign with, and which version of OS X to sign on.
> Is there any reason to believe we don't just pay the $100 to get a
> Mac Developer license at https://developer.apple.com/programs/mac/
> and then use Xcode to generate a cert per the link in Comment 0?

Not that I'm aware of.

But *shouldn't* the process be more complicated for institutions (like
Mozilla) than it is for individuals?  What if some random person
pretended to be "Mozilla"?
(Following up comment #29)

How does "Mozilla" get certs (signing or otherwise) from regular cert providers (like Thawte)?  Ones that represent the whole company.
(In reply to Steven Michaud from comment #30)
> (Following up comment #29)
> 
> How does "Mozilla" get certs (signing or otherwise) from regular cert
> providers (like Thawte)?  Ones that represent the whole company.

RelEng buys them. We're going to be looking at this this week.
(In reply to Alex Keybl [:akeybl] from comment #26)
> (In reply to Steven Michaud from comment #24)
> > > Are we able to get different keys from apple for our nightly and release builds?
> > 
> > I expect so, but I don't really know.  You might need to get them for
> > different Mozilla identities -- e.g. "Mozilla", "Mozilla Nightlies",
> > "Mozilla Try/Depend".
> > 
> > You (or someone) should contact Apple right away, to start the process of
> > getting at least one Apple ID for "Mozilla".  Alex might be able to suggest
> > who (or which department) at Apple you should contact.
> 
> Is there any reason to believe we don't just pay the $100 to get a Mac
> Developer license at https://developer.apple.com/programs/mac/ and then use
> Xcode to generate a cert per the link in Comment 0?

We need at least two certs as I indicated in comment 23 - we'll look into how we can do this.
(In reply to Steven Michaud from comment #29)
> But *shouldn't* the process be more complicated for institutions (like
> Mozilla) than it is for individuals?  What if some random person
> pretended to be "Mozilla"?

I'm not sure it's so much about verifying that we are Mozilla as it is about Apple having the ability to disable mischievous apps from being run remotely (with a $100 entrance fee and credit card information to nail the dev if need be).
(In reply to comment #33)

But surely it'd be very embarrassing to Apple, and to us, if they incorrectly identified someone as "Mozilla".  More embarrassing than if they identified someone incorrectly as "Steven Michaud" :-)
Now that I'm done with buildduty, this bug (and all others related to getting Mac builds signed) is my #1 priority.
I made some good strides yesterday:
- We now have a Developer ID to do manual test signing with. I intend to delete and create fresh ones to use in production when we're ready.
- It appears that installing Xcode 4.2 on our existing 10.6-based signing machine allows it to sign both the builds produced on 10.6+Xcode3 as well as the 10.7+Xcode4 ones. I'm still waiting for Steven to verify that the builds look OK on 10.8 (and maybe elsewhere).

Once everyone is happy with the manually signed builds, here's all the things we need to do before we can have signed Nightlies:
- Upgrade the other signing machine to Xcode 4.2
- Determine exactly how many Developer IDs we need, and generate the certificates
- Test out automated signing by pushing bug 723176 to try

After the above is done and looks OK we should be able to land bug 723176 on mozilla-central to get signed Nightlies. I would recommend that we spend a couple of days there, after which we should be able to land on aurora and beta simultaneously, as I don't think holding on Aurora exclusively for a few days will buy us much.

If anyone else wants access to the test builds you can find them here: http://people.mozilla.com/~bhearsum/samplebuilds/. Ping me on IRC or send me mail for the username and password.
(In reply to Ben Hearsum [:bhearsum] from comment #36)
> I made some good strides yesterday:
> - We now have a Developer ID to do manual test signing with. I intend to
> delete and create fresh ones to use in production when we're ready.
> - It appears that installing Xcode 4.2 on our existing 10.6-based signing
> machine allows it to sign both the builds produced on 10.6+Xcode3 as well as
> the 10.7+Xcode4 ones. I'm still waiting for Steven to verify that the builds
> look OK on 10.8 (and maybe elsewhere).

We had to tweak the CodeResources file we've been using, but after that I was able to sign both Firefox 12 and a Nightly with Xcode4.2+10.6 and have them validate on 10.8. Given that, I'm going to get to work on the rest of these steps:

> Once everyone is happy with the manually signed builds, here's all the
> things we need to do before we can have signed Nightlies:
> - Upgrade the other signing machine to Xcode 4.2
> - Determine exactly how many Developer IDs we need, and generate the
> certificates
> - Test out automated signing by pushing bug 723176 to try

Steven noticed that the Nightly I signed took awhile to validate, which is probably because it's got a lot of extra bits compared to 12. He suggested that we somewhat limit the files we sign, to speed that up. Steven, let's put that discussion in bug 723176, which is where the patch containing our CodeResources file is.
Depends on: 755367
> Steven noticed that the Nightly I signed took awhile to validate

At least once, signature validation took 10-15 seconds on both the nightly and FF 12.  But my results were inconsistent, and now I'm not able to reproduce them at all.

We should still keep this issue in mind.  And if I discover a way to reproduce it consistently I'll comment at bug 723176.  But in the meantime we've all got plenty of other things to do :-)

For the record, in my tests the signature validation took about 4 seconds on the nightly and about 3 seconds on FF 12.  Before each test I did the following:

1) Delete all copies of FF from the standard locations (e.g. the desktop and /Applications), then empty the trash.

2) Run the following command to rebuild the Launch Services database (watch out for line breaks):

/System/Library/Frameworks/CodeServices.framework/Frameworks/LaunchServices.framework/Support/lsregister -kill -r -domain user -domain local -domain system

3) Reboot the computer.
(Following up comment #39)

Note that (as best I can tell) signature validation only happens the first time you run a given copy of an app.  Some sort of flag must get written to the launch services database, and subsequent launches are much faster.
I just published a blog post summarizing what we've done, where we're at, and a timeline for getting everything finished: http://blog.mozilla.org/bhearsum/archives/287
What's the plan about developer-id-signing other Mozilla applications than Firefox?
No longer blocks: mountain-lion-compat, 730924
No longer depends on: 755367
If there is a plan for other apps, it should be discussed in a different bug than this one.
Blocks: 756830
I think this is a a dupe of bug 730924 at this point. bug 400296 tracked all the infrastructure required to sign Mac builds. bug 723176 has the patch that needs to land on mozilla-central (and aurora and beta). We have our Developer ID certificates, and nightlies are now signed (starting with http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2012-05-22-14-28-35-mozilla-central/firefox-15.0a1.en-US.mac.dmg). bug 723176 will track enabling on aurora and beta, when we're ready to.
Status: NEW → RESOLVED
Closed: 13 years ago
Resolution: --- → DUPLICATE
Product: mozilla.org → Release Engineering
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.