Open Bug 1928013 Opened 1 year ago Updated 1 year ago

Does the 132.0 update (via in-app update) lack apple gatekeeper notarization?

Categories

(Release Engineering :: Release Automation, defect, P3)

Tracking

(Not tracked)

People

(Reporter: freddy, Assigned: haik)

References

Details

Attachments

(2 files, 1 obsolete file)

We just got this email to security@mozilla.org:

Hello Firefox Security,

Regarding Firefox.app on macOS.
When updated from version 131.0.3 to 132.0 via the in-app update mechanism we
are seeing various components which seem to lack Apple Gatekeeper
notarization.

Specifically:
Firefox.app/Contents/Frameworks/ChannelPrefs.framework

and

Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework

This differs from the Firefox.app within the .Firefox 132.0.dmg
available from the Mozilla[.]org download site.

Is this the intended behavior of the app when updated via the in-app updater?

Thank you,

It would be prudent to investigate this. We do have in-app update QA for all releases, don't we?

Thank you :freddy for filing this bug!

This differs from the Firefox.app within the .Firefox 132.0.dmg available from the Mozilla[.]org download site.

Is this the intended behavior of the app when updated via the in-app updater?

No, this is not the expected behavior. We have tens of tasks called release-update-verify-firefox-macosx64-* in charge of detecting any discrepancy between a partial update (e.g. 131.0.3 to 132.0) and a complete one. They run at each release promotion.

I think we need more info to understand how the user ended up in this state. Let's follow up over email.

Assignee: nobody → jlorenzo

I did some sanity checking. I inspected the DMGs and the complete MARs (these are used for updates) from 132.0, 131.0.3, and 126.0. All of them appeared to be notarized. I have not checked the partial MARs yet (this is not trivial to do).

Some additional info would be helpful:

  • The output of spctl -a -vvv -t install for each of these frameworks
  • A copy of the entire Firefox.app bundle
  • Any specific errors or symptoms that are believed to be connected to this

Freddy - can you help us get this, or put us directly in touch with the reporter?

Flags: needinfo?(fbraun)

I inspected the DMGs and the complete MARs [...] I have not checked the partial MARs yet (this is not trivial to do).

The problem might not be in the files as we release them, but as a result of the update process itself. Does an update these days completely replace the original install directory/app? Or would it be possible that some injected/modified file survived a complete or partial update?

(In reply to Daniel Veditz [:dveditz] from comment #3)

I inspected the DMGs and the complete MARs [...] I have not checked the partial MARs yet (this is not trivial to do).

The problem might not be in the files as we release them, but as a result of the update process itself. Does an update these days completely replace the original install directory/app? Or would it be possible that some injected/modified file survived a complete or partial update?

Bytesized is best positioned to answer this question, but I would be very surprised if the update process modified these files directly. In the case of complete MARs the files are replaced fully. In the case of partial MARs, we apply binary patches.

Flags: needinfo?(bytesized)

When we use a partial MAR and patch the files, we check the hash of the file and, if it is as-expected, we apply the patch. If it is not, we fall back to a complete MAR. When we use a complete MAR, we read a file in the directory that basically tells us all the files that are part of the installation (though channel prefs and, I think, update settings are intentionally missing so that the update channel is preserved for RC builds). We basically delete all the files in the installation and replace them with the ones in the MAR.

Let me know if you have any further questions about this.

Flags: needinfo?(bytesized)
Flags: needinfo?(dveditz)

Freddy, another question for the bug submitters: was the computer offline at the time they verified the notarization status of the frameworks?

A screenshot of the results of the Apparency.app for Firefox.app after having been updated from 131.03 to 132.0 via the native in-app update mechanism.

OP here.
I'm an intermediate level macOS user with no coding ability.
I'll try my best to answer the questions in this thread and the ones that were forwarded to me by Frederik Braun [:freddy] in response to my initial emailed report.

MacOS 14.7
Firefox en-US 131.0.3 -> 132.0 via native in-app updater.
During the update and subsequent required relaunch of the app we were warned by BlockBlock.app [1] several times that firefox and some its components were not notarized.

2024-10-29 16:23:58.064392-0400 0x5915     Default     0x148e5              539    0    BlockBlock: [com.objective-see.blockblock:daemon] user says, 'allow', so allowing process="process":{"pid":1615,"name":"firefox","path":"/Applications/Firefox.app/Contents/MacOS/firefox","uid":502,"architecture":"unknown","arguments":["/Applications/Firefox.app/Contents/MacOS/firefox","-foreground"],"ppid":1,"rpid":1615,"ancestors":[1],"signing info (reported)":{"csFlags":570512129,"platformBinary":0,"signingID":"org.mozilla.firefox","teamID":"43AQ936H96","cdHash":"780946CBC1E9A372F6E43C4284253B3CD77FED34"},"signing info (computed)":{}}, item file path=(null), timestamp=2024-10-29 20:23:46 +0000, item binary=name=Firefox, object=/Applications/Firefox.app/Contents/MacOS/firefox


2024-10-29 16:25:47.829431-0400 0x5b8d     Default     0x157b1              539    0    BlockBlock: [com.objective-see.blockblock:daemon] user says, 'allow', so allowing process="process":{"pid":1669,"name":"firefox","path":"/Applications/Firefox.app/Contents/MacOS/firefox","uid":502,"architecture":"unknown","arguments":["/Applications/Firefox.app/Contents/MacOS/firefox"],"ppid":1,"rpid":1669,"ancestors":[1],"signing info (reported)":{"csFlags":570512129,"platformBinary":0,"signingID":"org.mozilla.firefox","teamID":"43AQ936H96","cdHash":"780946CBC1E9A372F6E43C4284253B3CD77FED34"},"signing info (computed)":{}}, item file path=(null), timestamp=2024-10-29 20:25:42 +0000, item binary=name=Firefox, object=/Applications/Firefox.app/Contents/MacOS/firefox


2024-10-29 16:26:02.396955-0400 0x61e9     Default     0x157bd              539    0    BlockBlock: [com.objective-see.blockblock:daemon] user says, 'allow', so allowing process="process":{"pid":1674,"name":"updater","path":"/Applications/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater","uid":502,"architecture":"Apple Silicon","arguments":["/Applications/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater","--openAppBundle","/Applications/Firefox.app","-foreground"],"ppid":1,"rpid":1674,"ancestors":[1],"signing info (reported)":{"csFlags":570503953,"platformBinary":0,"signingID":"org.mozilla.updater","teamID":"43AQ936H96","cdHash":"42207B568136ED516E7849404F9CDE2EFB998D82"},"signing info (computed)":{}}, item file path=(null), timestamp=2024-10-29 20:25:50 +0000, item binary=name=updater, object=/Applications/Firefox.app/Contents/MacOS/updater.app/Contents/MacOS/org.mozilla.updater

[1] https://objective-see.org/products/blockblock.html
[1] https://github.com/objective-see/BlockBlock

After being surprised by the above BlockBlock.app warnings, I used the GUI app Apparency.app [2] to examine the Gatekeeper status of the newly updated Firefox.app and saw that some components were listed as not being notarized as shown in the screenshot attached above.
Seeing what appeared to be errors, I decided to trash the newly updated Firefox.app and to re-install it by downloading a fresh copy of Firefox 132.0.dmg from mozilla.org.

[2] https://www.mothersruin.com/software/Apparency/

I thought that I had archived the seemingly broken Firefox.app prior to deleting/re-installing.
But, the archived binary app does not seem to exhibit the broken notarization that was seen following the update, at least not using the GUI tools that I am familiar with.
It is possible that I accidentally archived the freshly installed copy from the .dmg instead of the broken version.
When I say, broken version, it is important to note that macOS itself did not report any problems with the app in terms of Gatekeeper warnings etc.

So this may or may not be the file in question:

Firefox--1320_native_update_non-notarized_components.zip

Hello :timothy.b! Thank you for reporting this bug and for providing this additional information! Let's see if there's something that comes up.

(In reply to Timothy [:timothy.b] from comment #10)

Created attachment 9434392 [details]
Firefox_autoupdate_non-notarized_components.jpg

A screenshot of the results of the Apparency.app for Firefox.app after having been updated from 131.03 to 132.0 via the native in-app update mechanism.

It's interesting that 2 Frameworks have their signature busted here: Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework and Contents/Frameworks/ChannelPrefs.framework. All other signatures are valid. :bytesized, :bhearsum, would you happen to know anything that could lead to this state?

Flags: needinfo?(bytesized)
Flags: needinfo?(bhearsum)

(In reply to bhearsum@mozilla.com (:bhearsum) from comment #2)

[...]
Freddy - can you help us get this, or put us directly in touch with the reporter?

Clearing NI as OP got back to us. I'm also removing myself as the assignee because multiple people are looking into this bug.

Assignee: jlorenzo → nobody
Flags: needinfo?(fbraun)

(In reply to Haik Aftandilian [:haik] from comment #8)

[...]
The additional file is breaking the code signature of the bundle, but presumably macOS continues to let the application be launched. It is known to not be strict in cases where the bundle has already passed Gatekeeper and been verified.

That may explain the state :timothy.b was in ⬇️

(In reply to Timothy [:timothy.b] from comment #14)

[...]
When I say, broken version, it is important to note that macOS itself did not report any problems with the app in terms of Gatekeeper warnings etc.

Thank you for providing all of this information Timothy, it's very helpful!

One thing I think we can say fairly confidently is that this is not a widespread issue, seeing as it only came up because of BlockBlock and/or Apparency. (This doesn't mean we ought not to do anything about it necessarily, but absent any other evidence, it means it's not an emergency.)

(In reply to Johan Lorenzo [:jlorenzo] from comment #15)

Hello :timothy.b! Thank you for reporting this bug and for providing this additional information! Let's see if there's something that comes up.

(In reply to Timothy [:timothy.b] from comment #10)

Created attachment 9434392 [details]
Firefox_autoupdate_non-notarized_components.jpg

A screenshot of the results of the Apparency.app for Firefox.app after having been updated from 131.03 to 132.0 via the native in-app update mechanism.

It's interesting that 2 Frameworks have their signature busted here: Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework and Contents/Frameworks/ChannelPrefs.framework. All other signatures are valid. :bytesized, :bhearsum, would you happen to know anything that could lead to this state?

To be clear: I don't see any evidence that signatures have been broken. The attached screenshot shows that Gatekeeper believes ChannelPrefs.framework and UpdateSettings.framework are not notarized properly though, as I understand it.

However, on my own macOS machine I cannot repro this. Here's some output from codesign -vvv and spctl:

bhearsum-tdjgh6:o vpn$ for i in Firefox.app Firefox.app/Contents/Frameworks/ChannelPrefs.framework/ Firefox.app/Contents/MacOS/updater.app/ Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/; do echo "***** Checking signature on $i" && codesign -vvv $i && echo "***** Checking notarization on $i" && spctl -a -vvv -t install $i && echo; done;
***** Checking signature on Firefox.app
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/pingsender
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/pingsender
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libfreebl3.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libfreebl3.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/nmhproxy
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/nmhproxy
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libsoftokn3.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libsoftokn3.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/liblgpllibs.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/liblgpllibs.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libosclientcerts.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libosclientcerts.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozglue.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozglue.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/plugin-container.app
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/plugin-container.app
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/crashreporter.app
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/crashreporter.app
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozavutil.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozavutil.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libgkcodecs.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libgkcodecs.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/XUL
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/XUL
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/updater.app
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/updater.app
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libipcclientcerts.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libipcclientcerts.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libnss3.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libnss3.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libnssckbi.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libnssckbi.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozavcodec.dylib
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/libmozavcodec.dylib
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/minidump-analyzer
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/minidump-analyzer
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/media-plugin-helper.app
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/media-plugin-helper.app
--prepared:/Users/vpn/o/Firefox.app/Contents/Frameworks/ChannelPrefs.framework
--validated:/Users/vpn/o/Firefox.app/Contents/Frameworks/ChannelPrefs.framework
Firefox.app: valid on disk
Firefox.app: satisfies its Designated Requirement
***** Checking notarization on Firefox.app
Firefox.app: accepted
source=Notarized Developer ID
origin=Developer ID Application: Mozilla Corporation (43AQ936H96)

***** Checking signature on Firefox.app/Contents/Frameworks/ChannelPrefs.framework/
Firefox.app/Contents/Frameworks/ChannelPrefs.framework/: valid on disk
Firefox.app/Contents/Frameworks/ChannelPrefs.framework/: satisfies its Designated Requirement
***** Checking notarization on Firefox.app/Contents/Frameworks/ChannelPrefs.framework/
Firefox.app/Contents/Frameworks/ChannelPrefs.framework/: accepted
source=Notarized Developer ID
origin=Developer ID Application: Mozilla Corporation (43AQ936H96)

***** Checking signature on Firefox.app/Contents/MacOS/updater.app/
--prepared:/Users/vpn/o/Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework
--validated:/Users/vpn/o/Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework
Firefox.app/Contents/MacOS/updater.app/: valid on disk
Firefox.app/Contents/MacOS/updater.app/: satisfies its Designated Requirement
***** Checking notarization on Firefox.app/Contents/MacOS/updater.app/
Firefox.app/Contents/MacOS/updater.app/: accepted
source=Notarized Developer ID
origin=Developer ID Application: Mozilla Corporation (43AQ936H96)

***** Checking signature on Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/
Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/: valid on disk
Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/: satisfies its Designated Requirement
***** Checking notarization on Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/
Firefox.app/Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework/: accepted
source=Notarized Developer ID
origin=Developer ID Application: Mozilla Corporation (43AQ936H96)

Apparency also shows it as being notarized correctly for me (screenshot incoming).

Adding Haik to the needinfo, as this is more macOS-related than update related IMO.

Flags: needinfo?(bhearsum) → needinfo?(haftandilian)
Attachment #9434497 - Attachment is obsolete: true

Thank you for the report, Timothy.

Another question for you: in the Apparency settings, do you have the option Request Developer ID notarization ticket from Apple checked? If not, that might be having an effect here.

For some background, macOS Notarization is an additional step beyond codesigning. With codesigning, the file signatures are part of the file.

With notarization, it works differently. After submitting an app to Apple to be notarized (which checks for malware), the developer has the option to "staple" the resulting "ticket" to the app that was notarized. If the ticket is stapled, macOS can cryptographically confirm that the app was notarized without checking with Apple's servers at the time the app is first run. If the notarization ticket is not stapled, since the notarization status is not stored with the files, macOS has to check with Apple's servers to determine if the app is notarized. The check happens when the app is first launched.

One theory is that the frameworks are not individually stapled which may lead them to appear as un notarized for a period of time before the online check is done.

On a macOS 14.7 VM, I have been able to reproduce BlockBlock reporting the main Firefox executable was not notarized on a launch, but only once. I haven't been able to reproduce the frameworks appearing as not notarized in BlockBlock or Apparency.

Regarding integrity of the files, regardless of notarization status, if codesign -dvv <file> reports

...
Authority=Developer ID Application: Mozilla Corporation (43AQ936H96)
Authority=Developer ID Certification Authority
Authority=Apple Root CA
...

(the 43AQ936H96 string must match) then the file is from Mozilla.

I didn't find any issues with the file you provided.

Flags: needinfo?(haftandilian)
Assignee: nobody → haftandilian

(In reply to Johan Lorenzo [:jlorenzo] from comment #15)

It's interesting that 2 Frameworks have their signature busted here: Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework and Contents/Frameworks/ChannelPrefs.framework. All other signatures are valid. :bytesized, :bhearsum, would you happen to know anything that could lead to this state?

Given the comments that have been posted since then, I'm a little unclear on if this question is still relevant. We had some problems before where they'd end up being deleted, but I'm not sure about any ways that the signature could be invalid.

needinfo-ing @spohl in case he has anything to add.

Flags: needinfo?(bytesized) → needinfo?(spohl.mozilla.bugs)

(In reply to Robin Steuber (she/her) [:bytesized] from comment #22)

(In reply to Johan Lorenzo [:jlorenzo] from comment #15)

It's interesting that 2 Frameworks have their signature busted here: Contents/MacOS/updater.app/Contents/Frameworks/UpdateSettings.framework and Contents/Frameworks/ChannelPrefs.framework. All other signatures are valid. :bytesized, :bhearsum, would you happen to know anything that could lead to this state?

Given the comments that have been posted since then, I'm a little unclear on if this question is still relevant. We had some problems before where they'd end up being deleted, but I'm not sure about any ways that the signature could be invalid.

needinfo-ing @spohl in case he has anything to add.

The best theory is the one mentioned by Haik in comment 21. While there are various ways that signatures could become invalid, the issue here seems to be about notarization.

Flags: needinfo?(spohl.mozilla.bugs)
Flags: needinfo?(dveditz)
See Also: → 1928560

I filed bug 1928560 for my separate symptoms and marked my comments here "obsolete"

(In reply to Haik Aftandilian [:haik] from comment #9)

Freddy, another question for the bug submitters: was the computer offline at the time they verified the notarization status of the frameworks?

OP here. We were online unless there was an unknown network glitch/outage.

(In reply to Haik Aftandilian [:haik] from comment #21)

Thank you for the report, Timothy.

Another question for you: in the Apparency settings, do you have the option Request Developer ID notarization ticket from Apple checked? If ?> not, that might be having an effect here.

I believe "Request Developer ID notarization ticket from Apple` was checked during the time that Apparency.app was showing the components as being un-notarized.

The severity field is not set for this bug.
:bhearsum, could you have a look please?

For more information, please visit BugBot documentation.

Flags: needinfo?(bhearsum)

I'm marking this P3/S3 because it does not appear to be a widespread issue, and appears to have been worked around. I don't think we fully understand what happened, though.

Severity: -- → S3
Flags: needinfo?(bhearsum)
Priority: -- → P3
Component: Release Automation: Signing → Release Automation
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: