974570 - Sign MAR files on all platforms

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Brian R. Bondy [:bbondy]]

Currently MAR files are only signed on Windows. 

Doing a signmar -T firefox-30.0a1.en-US.win32.complete.mar
Reports 1 signature found.

Doing a signmar -T firefox-30.0a1.en-US.mac.complete.mar
Reports 0 signature found.

In preparation for the work tracked in bug 973933, could we sign MARs on all platforms?

Comment 1

[Brian R. Bondy [:bbondy]]

Please use the same keys as you do on Windows for this.
Checks for signatures are only enabled on Windows at the moment, but after the tracking bug dependencies land, it will be enabled on all platforms.

Comment 2

[Nick Thomas [:nthomas] (UTC+12)]

Looks like a Build Config issue:
* mar files are signed if MOZ_SIGN_PACKAGE_CMD is set
 http://hg.mozilla.org/mozilla-central/file/3762e1037b59/tools/update-packaging/Makefile.in#l66
* which depends on MOZ_EXTERNAL_SIGNING_FORMAT being set at 
 http://hg.mozilla.org/mozilla-central/file/3762e1037b59/toolkit/mozapps/installer/packager.mk#l562
* which is defined at 
 http://hg.mozilla.org/mozilla-central/file/3762e1037b59/toolkit/mozapps/installer/signing.mk#l24

Comment 3

[Brian R. Bondy [:bbondy]]

So I would just need to set 
> MOZ_EXTERNAL_SIGNING_FORMAT := signcode

For both Darwin and Linux?

Comment 4

[Nick Thomas [:nthomas] (UTC+12)]

<nthomas	catlee>: do you know what the right answer is on bug 974570 ? signcode doesn't seem right
       <catlee>: fix the makefiles to not depend indirectly on EXTERNAL_SIGNING_FORMAT

Comment 5

[Brian R. Bondy [:bbondy]]

Great thanks for the info, I'll test out a fix and submit a patch for review if it works.

Comment 6

[Brian R. Bondy [:bbondy]]

Attachment: signcmd.diff

Actually I'm really not confident that this is right so I'll go for feedback first before pushing to oak for testing.

Comment 7

[Chris AtLee [:catlee]]

Comment on attachment 8378660 [details] [diff] [review]
signcmd.diff

Review of attachment 8378660 [details] [diff] [review]:
-----------------------------------------------------------------

::: toolkit/mozapps/installer/packager.mk
@@ -563,1 @@
>  MOZ_SIGN_PACKAGE_CMD=$(MOZ_SIGN_CMD) $(foreach f,$(MOZ_EXTERNAL_SIGNING_FORMAT),-f $(f))

I'm not sure this is really the right way to do this, even if it works.

Maybe better is to modify http://hg.mozilla.org/mozilla-central/file/3762e1037b59/tools/update-packaging/Makefile.in#l48 or line 66 to act in if MOZ_SIGN_CMD is set, use it directly, like $(MOZ_SIGN_CMD) -f mar .../complete.mar

Comment 8

[Brian R. Bondy [:bbondy]]

Attachment: signcmd.diff

Like this?

Comment 9

[Chris AtLee [:catlee]]

Comment on attachment 8379016 [details] [diff] [review]
signcmd.diff

Review of attachment 8379016 [details] [diff] [review]:
-----------------------------------------------------------------

yeah, that looks more promising!

Comment 10

[Brian R. Bondy [:bbondy]]

OK thanks, I'll test it out and then mark it for review

Comment 11

[Brian R. Bondy [:bbondy]]

So for the nightly which happened last night (self serve ones aren't working) here:
ftp://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014-02-21-04-02-02-oak/

It looks like the linux mars are being signed now.
But there is no OSX mar available there, even know tbpl shows a successful nightly here:
https://tbpl.mozilla.org/?tree=Oak&rev=4102831c51b4

Comment 12

[Nick Thomas [:nthomas] (UTC+12)]

Maybe some caching issue, I see
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/02/2014-02-19-04-02-01-oak/firefox-30.0a1.en-US.mac.partial.20140215040201-20140219040201.mar
 http://ftp.mozilla.org/pub/mozilla.org/firefox/nightly/2014/02/2014-02-19-04-02-01-oak/firefox-30.0a1.en-US.mac.complete.mar

Comment 13

[Nick Thomas [:nthomas] (UTC+12)]

Oh whoops. Rev 4102831c51b4 was built on the 19th (links in comment #12). 1c875d373815 is the current tip, and failed in the mac build:
2014-02-21 05:04:19,208 - 1559ea42e979bc233cbb37c25ca8348e807b8532: processing ../../dist/update//firefox-30.0a1.en-US.mac.complete.mar on https://mac-signing3.build.scl1.mozilla.com:9100
2014-02-21 05:04:19,241 - 1559ea42e979bc233cbb37c25ca8348e807b8532: uploading for signing
2014-02-21 05:04:28,976 - 1559ea42e979bc233cbb37c25ca8348e807b8532: error uploading file for signing: 400 File too large

It's the regular 72M size, we're just hitting this limit set in the signing server:
 http://hg.mozilla.org/build/puppet/file/143dcde7e538/modules/signingserver/templates/signing.ini.erb#l29

Comment 14

[Brian R. Bondy [:bbondy]]

Thanks Nick!

Comment 15

[Chris AtLee [:catlee]]

Comment on attachment 8379016 [details] [diff] [review]
signcmd.diff

Review of attachment 8379016 [details] [diff] [review]:
-----------------------------------------------------------------

Let's get some eyes from the build system guys on this too.

Comment 16

[Brian R. Bondy [:bbondy]]

This is another bug I'd like to have tested before the nightly mars go out to everyone. Is that possible? I think you're doing this soon for rstrong in another bug.  I'd like to make sure that all platforms can apply the MAR files that are served when they are signed.

Comment 17

[Nick Thomas [:nthomas] (UTC+12)]

Sure, we can do that for mozilla-central if you want, probably better to do it on a separate day from rstrong. Does Oak not provide the testing you need though ? AFAIK we have the branch and update server set up to offer updates on the nightly-oak channel.

Comment 18

[Brian R. Bondy [:bbondy]]

Oak currently has the MAR verification changes on it, and I wanted to test it without that on it.  I could rebase it to m-c tip and then re-push everything I guess.  I think I'll do that to avoid making others do work, thanks for the suggestion.

Comment 19

[Nick Thomas [:nthomas] (UTC+12)]

Ah, the devil is in the details. Happy to do the update server lock/unlock for nightly whenever.

Comment 20

[Brian R. Bondy [:bbondy]]

FYI I tested this on Oak as per the above plan and this is working correctly.
I also verified that each full and partial MAR for each platform is being signed now.

I'll be pushing this out to mozilla-inbound when the tree opens back up.  So it should be on tomorrow's nightly.  Just a heads up in case there are any problems with updates being consumed.

Comment 21

[Brian R. Bondy [:bbondy]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/e9bf2f094cb2

Comment 22

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

https://hg.mozilla.org/mozilla-central/rev/e9bf2f094cb2

Comment 23

[[e-mail] (:bhearsum)]

Looks like MARs are getting signed on Mac and Linux now:
2014-03-13 08:03:11,718 - 02e7cda4da7f306dec98849c26b741c644e0b709: processing ../../dist/update//firefox-30.0a1.en-US.linux-i686.complete.mar on https://signing4.srv.releng.scl3.mozilla.com:9100
2014-03-13 08:03:11,995 - 02e7cda4da7f306dec98849c26b741c644e0b709: uploading for signing
2014-03-13 08:05:06,746 - e7672b9a3454a476e3cf36c1633d711b5a918a80: processing ../../dist/update/firefox-30.0a1.en-US.linux-i686.partial.20140312030201-20140313030202.mar on https://signing4.srv.releng.scl3.mozilla.com:9100
2014-03-13 08:05:06,955 - e7672b9a3454a476e3cf36c1633d711b5a918a80: uploading for signing

2014-03-13 04:14:56,770 - 8455ff6be21657e7701df3c6bf8ebd04a60cae11: processing ../../dist/update//firefox-30.0a1.en-US.mac.complete.mar on https://mac-signing3.build.scl1.mozilla.com:9100
2014-03-13 04:14:57,093 - 8455ff6be21657e7701df3c6bf8ebd04a60cae11: uploading for signing
2014-03-13 04:20:15,405 - 5e92078ee44eea470d925fa7b18835aa7574c553: processing ../../dist/update/firefox-30.0a1.en-US.mac.partial.20140312030201-20140313030202.mar on https://mac-signing2.srv.releng.scl3.mozilla.com:9100
2014-03-13 04:20:15,791 - 5e92078ee44eea470d925fa7b18835aa7574c553: uploading for signing

Comment 24

[Brian R. Bondy [:bbondy]]

Thanks for confirming Ben