Try update-verify needs to ignore `Contents/CodeResources` diffs
Categories
(Release Engineering :: Release Automation: Updates, defect)
Tracking
(firefox-esr78 unaffected, firefox82 unaffected, firefox83 unaffected, firefox84- wontfix, firefox115 fixed)
Tracking | Status | |
---|---|---|
firefox-esr78 | --- | unaffected |
firefox82 | --- | unaffected |
firefox83 | --- | unaffected |
firefox84 | - | wontfix |
firefox115 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: jcristau)
References
Details
(Keywords: intermittent-failure)
Attachments
(1 file)
Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=321727187&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/dutFgkiYT5iPAd2Yry6jiw/runs/0/artifacts/public/logs/live_backing.log
[task 2020-11-13T17:34:16.439Z] succeeded
[task 2020-11-13T17:34:16.439Z] calling QuitProgressUI
[task 2020-11-13T17:34:18.352Z] Comparing source/Firefox.app with target/Firefox.app...
[task 2020-11-13T17:34:18.352Z] Files only in source/Firefox.app:
[task 2020-11-13T17:34:18.352Z] Contents/CodeResources
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-11-13T17:34:18.352Z] TEST-UNEXPECTED-FAIL: [83.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 83.0b9.dmg vs. downloads/Firefox 84.0b1.dmg: 1
Comment 1•4 years ago
|
||
Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.
Comment 2•4 years ago
|
||
(In reply to Sebastian Hengst [:aryx] (needinfo on intermittent or backout) from comment #1)
Ben, can you take a look at this or forward the needinfo request? This might be blocking the release of 84.0b1 on Monday.
As far as I can tell, the try release build is not signed correctly (it's missing Contents/CodeResources, and possibly other issues). My first guess is that it's related to the Apple Silicon work that's happened recently. Aki, do you have any insight here?
Updated•4 years ago
|
Comment hidden (Intermittent Failures Robot) |
Comment 4•4 years ago
|
||
I dug into this a bunch this morning and here's what I found:
- Code signatures are broken when doing dep signing.
codesign -vvvv
shows these errors:
Firefox.app/: nested code is modified or invalid
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liblgpllibs.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libfreebl3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/firefox-bin
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/pingsender
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libsoftokn3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libplugin_child_interpose.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/plugin-container.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/XUL
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libosclientcerts.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozglue.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavutil.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/crashreporter.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libgraphitewasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnssckbi.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/updater.app
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libmozavcodec.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/libnss3.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/liboggwasm.dylib
file modified: /private/tmp/2020-11-16/Firefox.app/Contents/MacOS/minidump-analyzer
- Dep signing doesn't run
codesign -vvvv
before returning (probably because notarization is skipped), which means we publish a busted .app in taskcluster, which eventually results in the update verify errors shown in earlier comments here. We're actually lucky we caught them at all - update verify only noticed because the issue also results in Contents/CodeResources being missing from the package. If that wasn't the case, update verify would be green. - Resigning the exact same unsigned build with release keys works 100% fine, so I do not believe this will be an issues on Beta, Release, or ESR.
Comment 5•4 years ago
|
||
Given that this appears to be a Try-only issue, I'm dropping tracking status for this bug. We probably still want to get it fixed in a timely manner, though, to at least avoid false positives and noise when running simulations on Try.
Comment 6•4 years ago
|
||
I haven't found anything yet. Secretly hoping cross-compiled builds fixes this.
Assignee | ||
Comment 7•4 years ago
|
||
We have cross-compiled builds on central and beta now, is this still an issue?
Comment 8•4 years ago
|
||
We are still seeing the failure from comment 0:
[task 2020-12-02T18:03:03.788Z] Files only in source/Firefox.app:
[task 2020-12-02T18:03:03.788Z] Contents/CodeResources
[task 2020-12-02T18:03:03.788Z] TEST-UNEXPECTED-FAIL: differences found after update
[task 2020-12-02T18:03:03.789Z] TEST-UNEXPECTED-FAIL: [84.0 en-US complete] check_updates returned failure for Darwin_x86_64-gcc3-u-i386-x86_64 downloads/Firefox 84.0b7.dmg vs. downloads/Firefox 85.0b1.dmg: 1
Comment 9•4 years ago
•
|
||
Oh, duh.
It looks like Contents/CodeResources
is created by notarization, which makes sense since it only exists on the release-signed and notarized build. https://eclecticlight.co/2019/05/31/can-you-tell-whether-code-has-been-notarized/ :
You can also look inside the app: the notarization ‘ticket’ is seen as a file named CodeResources in the Contents folder. But that only applies to apps which have gone through the hardening and notarization process from new.
Contents/_CodeSignature/CodeResources
is created by signing. It exists on both the release-signed and dep-signed builds. So update-verify is dying because we're comparing a notarized release-signed app vs a non-notarized dep-signed app. I think we need to train update-verify on Try to ignore Contents/CodeResources
for mac.
Updated•4 years ago
|
Comment 10•4 years ago
|
||
(In reply to Aki Sasaki [:aki] (he/him) from comment #9)
I think we need to train update-verify on Try to ignore
Contents/CodeResources
for mac.
Nice sleuthing! Another thing we could do here is delete this file in the "from" build before applying the update (or doing the diff) when we're doing a try release.
Updated•3 years ago
|
Assignee | ||
Comment 12•1 years ago
|
||
Updated•1 years ago
|
Comment 13•1 years ago
|
||
Comment 14•1 years ago
|
||
bugherder |
Updated•1 years ago
|
Updated•1 year ago
|
Description
•