Closed Bug 1091668 Opened 5 years ago Closed 5 years ago

Generate and sign voucher for plugin-container

Categories

(Release Engineering :: General, defect)

x86_64
Linux
defect
Not set

Tracking

(firefox37 fixed, firefox38 fixed, firefox39 fixed)

RESOLVED FIXED
Tracking Status
firefox37 --- fixed
firefox38 --- fixed
firefox39 --- fixed

People

(Reporter: catlee, Assigned: mshal)

References

(Blocks 1 open bug)

Details

(Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/775] )

Attachments

(6 files, 2 obsolete files)

To support EME, we need to generate a voucher file using a script provided to us by Adobe, and then have this voucher signed.

I'm not sure if the script should be incorporated into the build system directly, or if the signing server should take care of the voucher generation and signing.
We discussed this on IRC a bit. The voucher file script seems to just be generating a hash of the text section, we could pretty easily run that as part of stage-package or something like that. The actual signing of the voucher should happen on the signing server, since it already has private keys.
assigning to me, per mtg with :coop just now. Not yet certain whats involved, still gathering requirements and necessary milestones
Assignee: nobody → bugspam.Callek
The basic flow here is that the build system will create this voucher, perhaps as part of packaging. The build system will then invoke the regular signing script with a new signing format on the voucher.

On the signing server side of things we need to implement the new signing format so that we return a PKCS7 signature. Exactly format is TBD, but the command to run will probably look something like this:

openssl smime -binary -sign -signer voucher1.pem -in voucher.bin

The result then gets sent back to the build system and included in the packaged build.
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/641]
Whiteboard: [kanban:engops:https://kanbanize.com/ctrl_board/6/641] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/775] [kanban:engops:https://kanbanize.com/ctrl_board/6/641]
Attached patch 0002-Make-sign-EME-voucher.patch (obsolete) — Splinter Review
Looking for some general feedback here:

1) Is there a better way to get the python dependencies (bitstring and pyasn1) in the virtualenv without importing the whole packages?

2) I made it as part of installer since the makensis.mk code was similar and also does signing. We could stick it in stage-package or elsewhere though if that makes more sense.

3) From #c4 it sounds like the actual signing will need to be different. I haven't tested that part yet anyway :)
Attachment #8516884 - Flags: feedback?(ted)
In case you want to try out patch 0002, here's the part that imports all the dependencies and the voucher script.

We still need to sort out whether it makes sense to put the voucher script in the tree, or if we need to download it via tooltool or something for licensing reasons (if we do, that might work around my virtualenv question).
cpearce, what do we need to do with the EME voucher after it gets signed? Is the voucher.bin file included anywhere as part of the package / installer / etc?
Flags: needinfo?(cpearce)
We haven't received an OK from Adobe to check their voucher generator into a public repo, so I'm making this bug moco-confidential until we receive their OK.
Blocks: EME, 1032660
Group: mozilla-employee-confidential
(In reply to Michael Shal [:mshal] from comment #7)
> cpearce, what do we need to do with the EME voucher after it gets signed? Is
> the voucher.bin file included anywhere as part of the package / installer /
> etc?

We need to stick the voucher in a well defined place, so that the GMP loading code (GMPChild.cpp specifically) can read that file when it starts up.

So please just write the voucher to a file which is installed some place where we can read that file at runtime, and I'll do the rest.
Flags: needinfo?(cpearce)
OK, I looked back over my notes, we're good to check in the voucher script to a public repo. I will follow up with Adobe about what license they want on the code...
Group: mozilla-employee-confidential
Got more details of how to generate the signature in the correct format. The proper cmdline is:

openssl smime -sign -in voucher.bin -signer signer.pem -md sha256 -binary -nodetach -outform DER -out voucher.sig

you can verify the signatures with

openssl smime -verify -in voucher.sig -inform DER

you may need to add -noverify there if you're testing with self-signed certificates
Mike is working on the build system parts, which align best with this bugs comments thus far, and has patches, I'll spin a new bug for signing-server support.
Assignee: bugspam.Callek → mshal
Depends on: 1094551
This should probably be a Build Config bug but I don't know if that will mess with your kanban stuff.
Comment on attachment 8516884 [details] [diff] [review]
0002-Make-sign-EME-voucher.patch

Review of attachment 8516884 [details] [diff] [review]:
-----------------------------------------------------------------

I don't see any simpler way around including the Python dependencies in-tree, it's not that big of a deal.

Do we know if this is going to be Windows-only?

I think this ought to happen as part of package staging via packager.mk, not just in the installer build.

Otherwise this general approach seems fine.

::: toolkit/mozapps/installer/windows/eme/makeeme.mk
@@ +6,5 @@
> +
> +installer::
> +	$(PYTHON) $(topsrcdir)/python/eme/gen-eme-voucher.py -input $(DIST)/bin/plugin-container.exe -output $(DIST)/bin/voucher.bin
> +ifdef MOZ_EXTERNAL_SIGNING_FORMAT
> +	$(MOZ_SIGN_CMD) $(foreach f,$(MOZ_EXTERNAL_SIGNING_FORMAT),-f $(f)) "$(DIST)/bin/voucher.bin"

This is probably not what we want, you'll probably want a specific signing format here.
Attachment #8516884 - Flags: feedback?(ted)
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #14)
> Do we know if this is going to be Windows-only?

We'll need to use a different voucher script on other platforms. Currently the script Adobe gave us works on COFF binaries, and obviously that won't work or whatever binary format other platforms use.

I'd expect we'd definitely have to do this for MacOSX, and probably have to do it for Linux too.
(In reply to Chris Pearce (:cpearce) from comment #15)
> I'd expect we'd definitely have to do this for MacOSX, and probably have to
> do it for Linux too.

Why wouldn't we have to do it for Linux, too?
(In reply to Henri Sivonen (:hsivonen) from comment #16)
> (In reply to Chris Pearce (:cpearce) from comment #15)
> > I'd expect we'd definitely have to do this for MacOSX, and probably have to
> > do it for Linux too.
> 
> Why wouldn't we have to do it for Linux, too?

I don't know how we'll do device binding on Linux. I'm not saying it's impossible, just that I don't know what will be acceptable to Adobe, given that anything on a Linux system can be patched.
(In reply to Chris Pearce (:cpearce) from comment #10)
> OK, I looked back over my notes, we're good to check in the voucher script
> to a public repo. I will follow up with Adobe about what license they want
> on the code...


Joe Steele from Adobe has confirmed via email that we should use the following license header on their voucher script:


Copyright 2014 Adobe Systems Incorporated. All Rights Reserved.
 
Adobe permits you to use, modify, and distribute this file in accordance 
with the terms of the Mozilla Public License, v 2.0 accompanying it.  If 
a copy of the MPL was not distributed with this file, You can obtain one 
at http://mozilla.org/MPL/2.0/.
Whiteboard: [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/775] [kanban:engops:https://kanbanize.com/ctrl_board/6/641] → [kanban:engops:https://mozilla.kanbanize.com/ctrl_board/6/775]
Comment on attachment 8516884 [details] [diff] [review]
0002-Make-sign-EME-voucher.patch

Review of attachment 8516884 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/installer/windows/eme/makeeme.mk
@@ +6,5 @@
> +
> +installer::
> +	$(PYTHON) $(topsrcdir)/python/eme/gen-eme-voucher.py -input $(DIST)/bin/plugin-container.exe -output $(DIST)/bin/voucher.bin
> +ifdef MOZ_EXTERNAL_SIGNING_FORMAT
> +	$(MOZ_SIGN_CMD) $(foreach f,$(MOZ_EXTERNAL_SIGNING_FORMAT),-f $(f)) "$(DIST)/bin/voucher.bin"

fwiw, I'm naming the format "emevoucher" over in Bug 1094551. Open to other suggestions though.
I'm splitting the imports of pysan1/bitstring/voucher script into separate patches for easier legal review. r?ted for the location of pyasn1 and r?gerv for licensing (BSD 2-clause)
Attachment #8516885 - Attachment is obsolete: true
Attachment #8520885 - Flags: review?(ted)
Attachment #8520885 - Flags: review?(gerv)
Import bitstring, MIT licensed.
Attachment #8520886 - Flags: review?(ted)
Attachment #8520886 - Flags: review?(gerv)
Import the EME voucher script. Adobe copyright, but MPL licensed per #c18.
Attachment #8520887 - Flags: review?(ted)
Attachment #8520887 - Flags: review?(gerv)
The actual hook in stage-package. I haven't been able to test the signing properly yet, and won't until bug 1094551 is fixed. However, I don't expect this patch to change much based on that if it needs to.
Attachment #8516884 - Attachment is obsolete: true
Attachment #8520891 - Flags: review?(ted)
Comment on attachment 8520885 [details] [diff] [review]
0001-Bug-1091668-Import-pyasn1-0.1.7.patch

Review of attachment 8520885 [details] [diff] [review]:
-----------------------------------------------------------------

No problem with this license.

Gerv
Attachment #8520885 - Flags: review?(gerv) → review+
Comment on attachment 8520887 [details] [diff] [review]
0003-Bug-1091668-Import-eme-voucher-script.patch

Review of attachment 8520887 [details] [diff] [review]:
-----------------------------------------------------------------

<sigh> What's wrong with the standard MPL 2.0 boilerplate? It says exactly the same thing that Adobe's modified version does.

I can imagine asking them to reconsider this won't help this process, so r=gerv, with sadness.

Gerv
Attachment #8520887 - Flags: review?(gerv) → review+
These bugs are necessary for vouching and sandboxing a third-party CDM.
Blocks: eme-m2
Attachment #8520886 - Flags: review?(gerv) → review+
FYI try is happy: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=1feb996507ea

(The reds are us sorting out some issues on the signing server side).

Note we still need to deploy the actual signing keys before this can land, but it's definitely ready for review.
Blocks: 1049272
No longer blocks: eme-m2
Attachment #8520885 - Flags: review?(ted) → review+
Attachment #8520886 - Flags: review?(ted) → review+
Comment on attachment 8520887 [details] [diff] [review]
0003-Bug-1091668-Import-eme-voucher-script.patch

Review of attachment 8520887 [details] [diff] [review]:
-----------------------------------------------------------------

It's a little worrying that this is a Python 3 script, in that nothing else we have in-tree is a Python 3 script. Do we have a working python3 binary on the builders?
Attachment #8520891 - Flags: review?(ted) → review+
Attachment #8520887 - Flags: review?(ted) → review+
(In reply to Ted Mielczarek [:ted.mielczarek] from comment #28)
> It's a little worrying that this is a Python 3 script, in that nothing else
> we have in-tree is a Python 3 script. Do we have a working python3 binary on
> the builders?

This concerned me as well - we don't have python3 on the builders. Fortunately the binary runs fine with python2.7, but since they are using python3, it may mean we get an updated script in the future that requires python3. I'm not sure it makes sense to go ahead and deploy python3 until we actually have a need for it, though.
Do we need to ask Adobe for a Python 2.7 version of the script, or can we make it work?
(In reply to Chris Pearce (:cpearce) from comment #30)
> Do we need to ask Adobe for a Python 2.7 version of the script, or can we
> make it work?

We execute the script using the $(PYTHON) make variable, which points to a python2.7 installation. This works for us fine now since the script doesn't use any python3 features. The concern would be that since they appear to be using python3 (based on the #!/usr/bin/env python3.3 in the script), they may introduce python3 specific features in the future, which would work fine for them but will require more work for us.
Comment on attachment 8520891 [details] [diff] [review]
0004-Bug-1091668-Make-sign-EME-voucher.patch

Review of attachment 8520891 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/installer/make-eme.mk
@@ +5,5 @@
> +include $(MOZILLA_DIR)/toolkit/mozapps/installer/signing.mk
> +
> +ifdef MOZ_SIGN_CMD
> +  ifeq ($(OS_ARCH),WINNT)
> +    MAKE_SIGN_EME_VOUCHER := $(PYTHON) $(topsrcdir)/python/eme/gen-eme-voucher.py -input $(DIST)/bin/plugin-container.exe -output $(DIST)/$(STAGEPATH)$(MOZ_PKG_DIR)/voucher.bin && \

Notwithstanding the fact that I hate that we grow the makefiles involved in packaging, this should be acting on the staged file, not the one in DIST/bin.
(In reply to Mike Hommey [:glandium] from comment #33)
> Comment on attachment 8520891 [details] [diff] [review]
> 0004-Bug-1091668-Make-sign-EME-voucher.patch
> 
> Review of attachment 8520891 [details] [diff] [review]:
> -----------------------------------------------------------------
> 
> ::: toolkit/mozapps/installer/make-eme.mk
> > +    MAKE_SIGN_EME_VOUCHER := $(PYTHON) $(topsrcdir)/python/eme/gen-eme-voucher.py -input $(DIST)/bin/plugin-container.exe -output $(DIST)/$(STAGEPATH)$(MOZ_PKG_DIR)/voucher.bin && \
> 
> Notwithstanding the fact that I hate that we grow the makefiles involved in
> packaging, this should be acting on the staged file, not the one in DIST/bin.

We explicitly wanted it to act on the post-signed plugin-container.exe.
(In reply to Michael Shal [:mshal] from comment #29)
> This concerned me as well - we don't have python3 on the builders.
> Fortunately the binary runs fine with python2.7, but since they are using
> python3, it may mean we get an updated script in the future that requires
> python3. I'm not sure it makes sense to go ahead and deploy python3 until we
> actually have a need for it, though.

For clarity, could you change the shebang line so as not to be confusing? If we get an update to the script in the future we'll just have to verify that it continues to work with Python 2.7.
Reopening to address #c33 and #c36
Status: RESOLVED → REOPENED
Resolution: FIXED → ---
Feel free to land a shebang change with rs=me.
Depends on: 1104997
Were you looking to generate the voucher from the package-staged or installer-staged file? (dist/firefox/plugin-container.exe or installer-stage/core/plugin-container.exe?)

When I tested this before, the voucher generated is the same whether the underlying plugin-container.exe is signed or not. I don't mind updating it either way, but I don't think it will change the result.
Flags: needinfo?(mh+mozilla)
Flags: needinfo?(bugspam.Callek)
I think this is what you're looking for? It should generate the voucher on the signed plugin-container.exe.

I also changed the #! python per ted's request, and changed $(topsrcdir) to $(MOZILLA_DIR) for bug 1104997 (if that lands first I'll just rebase).

try: https://treeherder.mozilla.org/ui/#/jobs?repo=try&revision=145633270439
Attachment #8529285 - Flags: review?(mh+mozilla)
I would like to write a gtest to test that we can retrieve the generated voucher at runtime, however the voucher is not present in $objdir/dist/bin once `./mach build` has completed locally, so if I wrote a test it would fail locally. Is there a way to run the voucher generator locally and install the (unsigned) voucher in $objdir/dist/bin so that I can have the test past locally at least too?
Comment on attachment 8529285 [details] [diff] [review]
0001-Bug-1091668-generate-voucher-on-signed-plugin-contai.patch

Review of attachment 8529285 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/installer/make-eme.mk
@@ +6,5 @@
>  
>  ifdef MOZ_SIGN_CMD
>    ifeq ($(OS_ARCH),WINNT)
> +    # Note: CWD is DIST
> +    MAKE_SIGN_EME_VOUCHER := $(PYTHON) $(MOZILLA_DIR)/python/eme/gen-eme-voucher.py -input $(STAGEPATH)$(MOZ_PKG_DIR)/plugin-container.exe -output $(STAGEPATH)$(MOZ_PKG_DIR)/voucher.bin && \

IS it possible to call it plugin-container.voucher to match the naming convention used by the GMP vouchers? Thanks.
(In reply to Chris Pearce (:cpearce) from comment #42)
> IS it possible to call it plugin-container.voucher to match the naming
> convention used by the GMP vouchers? Thanks.

From a releng-signing-server perspective there is no problem with this. I *will* need to update the signing server config to allow the new file extension as "ok" though, if we go that route.
If it's not a big hassle, then yes please.
Comment on attachment 8529285 [details] [diff] [review]
0001-Bug-1091668-generate-voucher-on-signed-plugin-contai.patch

Review of attachment 8529285 [details] [diff] [review]:
-----------------------------------------------------------------

::: python/eme/gen-eme-voucher.py
@@ +1,1 @@
> +#!/usr/bin/env python

why not python2.7?
Attachment #8529285 - Flags: review?(mh+mozilla) → review+
Flags: needinfo?(mh+mozilla)
(In reply to Chris Pearce (:cpearce) from comment #41)
> I would like to write a gtest to test that we can retrieve the generated
> voucher at runtime, however the voucher is not present in $objdir/dist/bin
> once `./mach build` has completed locally, so if I wrote a test it would
> fail locally. Is there a way to run the voucher generator locally and
> install the (unsigned) voucher in $objdir/dist/bin so that I can have the
> test past locally at least too?

gtests are due to move out of make check in a separate test suite. Some day. It doesn't strike me as problematic to change gtest to run on a packaged build (that is, after make package).

Alternatively, you could run an xpcshell test or a mochitest.
(In reply to Justin Wood (:Callek) from comment #43)
> (In reply to Chris Pearce (:cpearce) from comment #42)
> > IS it possible to call it plugin-container.voucher to match the naming
> > convention used by the GMP vouchers? Thanks.
> 
> From a releng-signing-server perspective there is no problem with this. I
> *will* need to update the signing server config to allow the new file
> extension as "ok" though, if we go that route.

Actually, whatever, we can run with the existing voucher.bin name. It doesn't really matter at this stage what the file is called.
(In reply to Mike Hommey [:glandium] from comment #45)
> @@ +1,1 @@
> > +#!/usr/bin/env python
> 
> why not python2.7?

Is there a reason to do that? I just used #!/usr/bin/env python because that matches most of the rest of the tree (~660 files), though a good chunk (~115) use #!/usr/bin/python instead. I don't see any that call out python2.7 specifically.
Seamonkey windows builders have both python2.7 and python, and python is 2.6, not 2.7. Not that I think it matters much for this specific script.
Flags: needinfo?(bugspam.Callek)
https://hg.mozilla.org/mozilla-central/rev/0db3ba4b70ab
Status: REOPENED → RESOLVED
Closed: 5 years ago5 years ago
Resolution: --- → FIXED
This regressed somehow; I filed bug 1123990.
Depends on: 1135746
See Also: → 1265272
Component: General Automation → General
You need to log in before you can comment on or make changes to this bug.