1677171 - Try update-verify needs to ignore `Contents/CodeResources` diffs

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Updates, defect)

Description

[Treeherder Bug Filer]

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=321727187&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dutFgkiYT5iPAd2Yry6jiw/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-11-13T17:34:16.439Z] succeeded
[task 2020-11-13T17:34:16.439Z] calling QuitProgressUI
[task 2020-11-13T17:34:18.352Z] Comparing source/Firefox.app with target/Firefox.app...
[task 2020-11-13T17:34:18.352Z] Files only in source/Firefox.app:
[task 2020-11-13T17:34:18.352Z]   Contents/CodeResources
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: [83.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 83.0b9.dmg vs. downloads/Firefox 84.0b1.dmg: 1

Comment 1

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #1)

Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.

As far as I can tell, the try release build is not signed correctly (it's missing Contents/CodeResources, and possibly other issues). My first guess is that it's related to the Apple Silicon work that's happened recently. Aki, do you have any insight here?

Comment 3

[Intermittent Failures Robot]

1 failures in 4040 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• osx-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1677171&startday=2020-11-09&endday=2020-11-15&tree=all

Comment 4

[[e-mail] (:bhearsum)]

I dug into this a bunch this morning and here's what I found:

• Code signatures are broken when doing dep signing. codesign -vvvv shows these errors:

Firefox.app/: nested code is modified or invalid
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liblgpllibs.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libfreebl3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/firefox-bin
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/pingsender
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libsoftokn3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libplugin_child_interpose.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/plugin-container.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/XUL
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libosclientcerts.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozglue.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavutil.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/crashreporter.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libgraphitewasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnssckbi.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/updater.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavcodec.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnss3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liboggwasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/minidump-analyzer

• Dep signing doesn't run codesign -vvvv before returning (probably because notarization is skipped), which means we publish a busted .app in taskcluster, which eventually results in the update verify errors shown in earlier comments here. We're actually lucky we caught them at all - update verify only noticed because the issue also results in Contents/CodeResources being missing from the package. If that wasn't the case, update verify would be green.
• Resigning the exact same unsigned build with release keys works 100% fine, so I do not believe this will be an issues on Beta, Release, or ESR.

Comment 5

[Ryan VanderMeulen [:RyanVM]]

Given that this appears to be a Try-only issue, I'm dropping tracking status for this bug. We probably still want to get it fixed in a timely manner, though, to at least avoid false positives and noise when running simulations on Try.

Comment 6

[Aki Sasaki (not active)]

I haven't found anything yet. Secretly hoping cross-compiled builds fixes this.

Comment 7

[Julien Cristau [:jcristau]]

We have cross-compiled builds on central and beta now, is this still an issue?

Comment 8

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

We are still seeing the failure from comment 0:

[task 2020-12-02T18:03:03.788Z] Files only in source/Firefox.app:
[task 2020-12-02T18:03:03.788Z]   Contents/CodeResources
[task 2020-12-02T18:03:03.788Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-12-02T18:03:03.789Z] TEST-UNEXPECTED-FAIL: [84.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 84.0b7.dmg vs. downloads/Firefox 85.0b1.dmg: 1

Comment 9

[Aki Sasaki (not active)]

Oh, duh.
It looks like Contents/CodeResources is created by notarization, which makes sense since it only exists on the release-signed and notarized build. https://eclecticlight.co/2019/05/31/can-you-tell-whether-code-has-been-notarized/ :

You can also look inside the app: the notarization ‘ticket’ is seen as a file named CodeResources in the Contents folder. But that only applies to apps which have gone through the hardening and notarization process from new.

Contents/_CodeSignature/CodeResources is created by signing. It exists on both the release-signed and dep-signed builds. So update-verify is dying because we're comparing a notarized release-signed app vs a non-notarized dep-signed app. I think we need to train update-verify on Try to ignore Contents/CodeResources for mac.

Comment 10

[[e-mail] (:bhearsum)]

(In reply to Aki Sasaki [:aki] (he/him) from comment #9)

I think we need to train update-verify on Try to ignore Contents/CodeResources for mac.

Nice sleuthing! Another thing we could do here is delete this file in the "from" build before applying the update (or doing the diff) when we're doing a try release.

Comment 12

[Julien Cristau [:jcristau]]

Attachment: Bug 1677171 - update-verify: silence spurious errors on mac staging releases. r?#releng

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/786b01043005
update-verify: silence spurious errors on mac staging releases. r=releng-reviewers,bhearsum

Comment 14

[Cristian Tuns]

https://hg.mozilla.org/mozilla-central/rev/786b01043005