1375904 - Nightly on macOS loses code signature after update

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Jonathan Rudenberg]

User Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.12; rv:56.0) Gecko/20100101 Firefox/56.0
Build ID: 20170623151754

Steps to reproduce:

I installed Nightly 56.0a1 2017-06-22 from the macOS disk image release and then did an in-app update to 2017-06-23.


Actual results:

Before running the update:

$ codesign -dvv /Applications/FirefoxNightly.app
Executable=/Volumes/Nightly/FirefoxNightly.app/Contents/MacOS/firefox
Identifier=org.mozilla.nightly
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20200 size=243 flags=0x0(none) hashes=5+3 location=embedded
Library validation warning=OS X SDK version before 10.9 does not support Library Validation
Signature size=8537
Authority=Developer ID Application: Mozilla Corporation (43AQ936H96)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Jun 22, 2017, 06:33:12
Info.plist entries=21
TeamIdentifier=43AQ936H96
Sealed Resources version=2 rules=12 files=91
Internal requirements count=1 size=188

After running the delta update:

$ codesign -dvv /Applications/FirefoxNightly.app
/Applications/FirefoxNightly.app: code object is not signed at all


Expected results:

The app should remain signed after running an update.

Comment 1

[Alex Gaynor [:Alex_Gaynor]]

I think this is likely to be related to/caused by bug 1267425.

Comment 2

[Kim Moir [:kmoir] ET]

I reproduced the problem
1:48 PM also, the mar appears not to be signed
Kims-MacBook-Pro:Downloads kmoir$  codesign -dvv ~/Downloads/Firefox-mozilla-central-56.0a1-macosx64-en-US-20170622100314-20170623151754.partial.mar
/Users/kmoir/Downloads/Firefox-mozilla-central-56.0a1-macosx64-en-US-20170622100314-20170623151754.partial.mar: code object is not signed at all
1:48 PM that my nightly downloaded for the latest update, not sure if that is expected
1:51 PM 
<aki> Aki Sasaki partial mar would be funsize
1:54 PM 
<kmoir> Kim Moir I'm looking at funsize code, see what could happen
1:54 PM but the expectation is that the mars should be signed, correct?
1:54 PM 
<aki> Aki Sasaki yes
1:54 PM oh
1:55 PM this may have to do with tc indexes. we probably want the repackaged dmg in the index, not the unsigned dmg
1:55 PM 
<kmoir> Kim Moir hmm, okay will look'
1:55 PM 
<aki> Aki Sasaki the build is probably pushing to the index funsize is looking at, rather than the repackage task
1:58 PM 
<Callek> that may be, though I have no idea why it wouldn't sign a partial mar...
1:59 PM kmoir: aki: can one of you get a bug on file, I can probably fix the funsize routing...
1:59 PM 
<aki> Aki Sasaki +1, i may be wrong about why it's not signed
2:00 PM 
<kmoir> Kim Moir

Comment 3

[Justin Wood (:Callek)]

Attachment: Bug 1375904 - Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed.

Review commit: https://reviewboard.mozilla.org/r/152254/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/152254/

Comment 4

[Aki Sasaki (not active)]

Comment on attachment 8880886 [details]
Bug 1375904 - Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed.

https://reviewboard.mozilla.org/r/152254/#review157290

::: taskcluster/taskgraph/transforms/signing.py:135
(Diff revision 1)
>              'run-on-projects': dep_job.attributes.get('run_on_projects'),
>              'treeherder': treeherder,
>              'routes': job.get('routes', []),
>          }
>  
> -        if job.get('use-funsize-route', False):
> +        if 'macosx' not in dep_job.attributes.get('build_platform') and \

i think we'll have to add windows to the list to exclude at some point, since we'll index the signed-repackage task. that can wait til windows is ready, though.

Comment 5

[Justin Wood (:Callek)]

Comment on attachment 8880886 [details]
Bug 1375904 - Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/152254/diff/1-2/

Comment 6

[Justin Wood (:Callek)]

Comment on attachment 8880886 [details]
Bug 1375904 - Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed.

https://reviewboard.mozilla.org/r/152254/#review157292

After IRC chat, I updated this to remove *adding* funsize routes to the repackage task until we properly test and validate the partials it generates.

https://github.com/mozilla-releng/funsize/blob/544fc0d42311e63673f949cd0fc2815880158cbf/funsize/worker.py#L86  makes me think we'll need funsize changes anyway.

Comment 7

[Justin Wood (:Callek)]

Comment on attachment 8880886 [details]
Bug 1375904 - Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed.

Review request updated; see interdiff: https://reviewboard.mozilla.org/r/152254/diff/2-3/

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/d962ecbc3fd1
Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed. r=aki

Comment 9

[Kim Moir [:kmoir] ET]

I added balrog rule 606 to disable updates for macosx nightlies until this issue is resolved

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/92dc60b522d8
Fix funsize routes so we activate funsize on the dmg tasks, so partials should end up being signed. r=aki a=callek

Comment 11

[Wes Kocher (:KWierso) (Not reading bugmail; email directly if needed)]

Triggered mac nightly on https://hg.mozilla.org/mozilla-central/rev/92dc60b522d8 

I need to head out on pto in a few minutes to finish prepping for the all hands, so hopefully this works as expected and you can take things from here. :)

Comment 12

[Kim Moir [:kmoir] ET]

I'll verify on the nightlytest channel before enabling updates again

Comment 13

[Kim Moir [:kmoir] ET]

So now I'm getting a complete mar but the mar is not signed
Kims-MacBook-Pro:Downloads kmoir$ codesign -dvv firefox-56.0a1.en-US.mac.complete.mar
firefox-56.0a1.en-US.mac.complete.mar: code object is not signed at all

Comment 14

[Kim Moir [:kmoir] ET]

And of course, the entire app is not signed either after updating
Kims-MacBook-Pro:Applications kmoir$ codesign -dvv /Applications/FirefoxNightly.app/
/Applications/FirefoxNightly.app/: code object is not signed at all

Comment 15

[Kim Moir [:kmoir] ET]

So earlier I didn't know how to verify mar files.  Now I do from catlee's note on the email thread

----
FWIW, using `codesign` won't properly detect signatures on mar files. You need to use a tool like `signmar` or `mar` from mardor:

https://github.com/mozilla/build-mar

I think the command line would be something like this:
`mar -k :mozilla-nightly -v partial.mar`

---
so I tried this on the files I had from testing

complete from today's testing updating from the nightly test channel 

Kims-MacBook-Pro:Downloads kmoir$ mar -k :mozilla-nightly -v firefox-56.0a1.en-US.mac.complete.mar 
Verification OK

Other mar was from when we were serving partials 
Kims-MacBook-Pro:Downloads kmoir$ mar -k :mozilla-nightly -v Firefox-mozilla-central-56.0a1-macosx64-en-US-20170622100314-20170623151754.partial.mar
Verification OK

---

As a sanity check, I installed 55.0b3 and ran codesign -dvv on the app and then updated to b4 and ran it again and the app is signed both before and after.  Just to verify that this wasn't a problem before we switched to tc as a sanity check.

I also installed a new nightly from here just to check again
https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-job_type_symbol=Nr&selectedJob=109587414
set app.update.log to true in about:config
exited the browser

set the update channel to nightlytest
Kims-MacBook-Pro:Applications kmoir$  cd FirefoxNightly.app/
Kims-MacBook-Pro:FirefoxNightly.app kmoir$ cat Contents/Resources/defaults/pref/channel-prefs.js
/* This Source Code Form is subject to the terms of the Mozilla Public
 * License, v. 2.0. If a copy of the MPL was not distributed with this
 * file, You can obtain one at http://mozilla.org/MPL/2.0/. */

pref("app.update.channel", "nightlytest");

opened the browser, and looked at the browser console for the mar that is downloaded and downloaded it locally to test
 wget https://archive.mozilla.org/pub/firefox/nightly/2017/06/2017-06-24-10-01-12-mozilla-central/firefox-56.0a1.en-US.mac.complete.mar
Resolving archive.mozilla.org... 54.230.118.82
Connecting to archive.mozilla.org|54.230.118.82|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 59784683 (57M) [application/octet-stream]
Saving to: ‘firefox-56.0a1.en-US.mac.complete.mar.1’

firefox-56.0a1.en-US.mac.complete.mar.1                    100%[========================================================================================================================================>]  57.01M  2.79MB/s    in 21s     

2017-06-24 21:22:59 (2.75 MB/s) - ‘firefox-56.0a1.en-US.mac.complete.mar.1’ saved [59784683/59784683]

Kims-MacBook-Pro:Downloads kmoir$  mar -k :mozilla-nightly -v firefox-56.0a1.en-US.mac.complete.mar.1
Verification OK

ran the code verification tool locally
Kims-MacBook-Pro:Applications kmoir$  codesign -dvv /Applications/FirefoxNightly.app/
Executable=/Applications/FirefoxNightly.app/Contents/MacOS/firefox
Identifier=org.mozilla.nightly
Format=app bundle with Mach-O thin (x86_64)
CodeDirectory v=20200 size=243 flags=0x0(none) hashes=5+3 location=embedded
Signature size=8536
Authority=Developer ID Application: Mozilla Corporation (43AQ936H96)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
Timestamp=Jun 23, 2017, 11:43:50 AM
Info.plist entries=21
TeamIdentifier=43AQ936H96
Sealed Resources version=2 rules=12 files=91
Internal requirements count=1 size=188

restarted the browser to install the update

and looked at ran the codesign tool against the updated nightly

Kims-MacBook-Pro:Applications kmoir$  codesign -dvv /Applications/FirefoxNightly.app/
/Applications/FirefoxNightly.app/: code object is not signed at all

Comment 16

[Carsten Book [:Tomcat]]

https://hg.mozilla.org/mozilla-central/rev/d962ecbc3fd1

Comment 17

[Justin Wood (:Callek)]

Reopening because the landed patch didn't actually fix this (like I hoped).  (mainly reopen instead of new bug because this bug # is tracked elsewhere)

I'll use a new bug for the code changes to fix though, but blocking this one.

Comment 18

[Kim Moir [:kmoir] ET]

Can we close this bug now, I notice that nightlies are enabled in balrog again.

I tested the update progress today and it worked great, thanks Callek!

Comment 20

[Chris AtLee [:catlee]]

I think we can set updates at 100% and close this bug now.

Comment 21

[Justin Wood (:Callek)]

This went live about 2 hours ago.

Comment 22

[Frank Burleigh]

I believe I got a normal update June 30, but got none on July 1 and 2.

Comment 23

[Firefox Bug Husbandry Bot]

Removing leave-open keyword from resolved bugs, per :sylvestre.