Closed Bug 1677171 Opened 4 years ago Closed 1 years ago

Try update-verify needs to ignore `Contents/CodeResources` diffs

Categories

(Release Engineering :: Release Automation: Updates, defect)

defect

Tracking

(firefox-esr78 unaffected, firefox82 unaffected, firefox83 unaffected, firefox84- wontfix, firefox115 fixed)

RESOLVED FIXED
Tracking Status
firefox-esr78 --- unaffected
firefox82 --- unaffected
firefox83 --- unaffected
firefox84 - wontfix
firefox115 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: jcristau)

References

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=321727187&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dutFgkiYT5iPAd2Yry6jiw/runs/0/artifacts/public/logs/live_backing.log


[task 2020-11-13T17:34:16.439Z] succeeded
[task 2020-11-13T17:34:16.439Z] calling QuitProgressUI
[task 2020-11-13T17:34:18.352Z] Comparing source/Firefox.app with target/Firefox.app...
[task 2020-11-13T17:34:18.352Z] Files only in source/Firefox.app:
[task 2020-11-13T17:34:18.352Z]   Contents/CodeResources
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: [83.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 83.0b9.dmg vs. downloads/Firefox 84.0b1.dmg: 1

Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.

Severity: S4 → --
Flags: needinfo?(bhearsum)
Priority: P5 → --

(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #1)

Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.

As far as I can tell, the try release build is not signed correctly (it's missing Contents/CodeResources, and possibly other issues). My first guess is that it's related to the Apple Silicon work that's happened recently. Aki, do you have any insight here?

Flags: needinfo?(bhearsum) → needinfo?(aki)

I dug into this a bunch this morning and here's what I found:

  • Code signatures are broken when doing dep signing. codesign -vvvv shows these errors:
Firefox.app/: nested code is modified or invalid
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liblgpllibs.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libfreebl3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/firefox-bin
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/pingsender
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libsoftokn3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libplugin_child_interpose.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/plugin-container.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/XUL
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libosclientcerts.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozglue.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavutil.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/crashreporter.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libgraphitewasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnssckbi.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/updater.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavcodec.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnss3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liboggwasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/minidump-analyzer
  • Dep signing doesn't run codesign -vvvv before returning (probably because notarization is skipped), which means we publish a busted .app in taskcluster, which eventually results in the update verify errors shown in earlier comments here. We're actually lucky we caught them at all - update verify only noticed because the issue also results in Contents/CodeResources being missing from the package. If that wasn't the case, update verify would be green.
  • Resigning the exact same unsigned build with release keys works 100% fine, so I do not believe this will be an issues on Beta, Release, or ESR.
Summary: Perma macOS update-verify TEST-UNEXPECTED-FAIL: [83.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 83.0b9.dmg vs. downloads/Firefox 84.0b1.dmg: 1 → signing x86_64+aarch64 universal builds with dep certs is broken

Given that this appears to be a Try-only issue, I'm dropping tracking status for this bug. We probably still want to get it fixed in a timely manner, though, to at least avoid false positives and noise when running simulations on Try.

I haven't found anything yet. Secretly hoping cross-compiled builds fixes this.

Flags: needinfo?(aki)

We have cross-compiled builds on central and beta now, is this still an issue?

Flags: needinfo?(aryx.bugmail)

We are still seeing the failure from comment 0:

[task 2020-12-02T18:03:03.788Z] Files only in source/Firefox.app:
[task 2020-12-02T18:03:03.788Z]   Contents/CodeResources
[task 2020-12-02T18:03:03.788Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-12-02T18:03:03.789Z] TEST-UNEXPECTED-FAIL: [84.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 84.0b7.dmg vs. downloads/Firefox 85.0b1.dmg: 1
Flags: needinfo?(aryx.bugmail) → needinfo?(aki)

Oh, duh.
It looks like Contents/CodeResources is created by notarization, which makes sense since it only exists on the release-signed and notarized build. https://eclecticlight.co/2019/05/31/can-you-tell-whether-code-has-been-notarized/ :

You can also look inside the app: the notarization ‘ticket’ is seen as a file named CodeResources in the Contents folder. But that only applies to apps which have gone through the hardening and notarization process from new.

Contents/_CodeSignature/CodeResources is created by signing. It exists on both the release-signed and dep-signed builds. So update-verify is dying because we're comparing a notarized release-signed app vs a non-notarized dep-signed app. I think we need to train update-verify on Try to ignore Contents/CodeResources for mac.

Flags: needinfo?(aki)
Summary: signing x86_64+aarch64 universal builds with dep certs is broken → Try update-verify needs to ignore `Contents/CodeResources` diffs

(In reply to Aki Sasaki [:aki] (he/him) from comment #9)

I think we need to train update-verify on Try to ignore Contents/CodeResources for mac.

Nice sleuthing! Another thing we could do here is delete this file in the "from" build before applying the update (or doing the diff) when we're doing a try release.

Severity: -- → S4
Duplicate of this bug: 1822136
Assignee: nobody → jcristau
Status: NEW → ASSIGNED
Pushed by jcristau@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/786b01043005 update-verify: silence spurious errors on mac staging releases. r=releng-reviewers,bhearsum
Status: ASSIGNED → RESOLVED
Closed: 1 years ago
Resolution: --- → FIXED
Regressed by: 1836851
No longer regressed by: 1836851
Regressions: 1836851
See Also: → 1874207
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: