1799332 - Code signing failure due to wrong version of channel-prefs.js after update (macOS Ventura)

Status: RESOLVED FIXED

(Toolkit :: Application Update, defect, P2)

Description

[Gian-Carlo Pascutto [:gcp]]

Firefox Nightly, macOS 12.6 -> Ventura 13.0 update.

Nightly no longer launches with an error "FireforNightly" is damaged and can't be opened. You should move it to the Bin.

I see a similar behavior as in bug 1781111 but this isn't mozregression, it's my main browser install.

Comment 1

[Steven Michaud [:smichaud] (Retired)]

Which nightly, exactly? Does it happen with all nightlies?

Comment 2

[Steven Michaud [:smichaud] (Retired)]

I just tried downloading the latest Firefox Nightly from https://archive.mozilla.org (https://archive.mozilla.org/pub/firefox/nightly/2022/11/2022-11-05-21-53-50-mozilla-central/firefox-108.0a1.en-US.mac.dmg), installing it to /Applications, then running it (by double-clicking on it). I had no problems.

This was on a fresh install of macOS 13 (build 22A380, the current release), on an Intel Mac. I'm sure I've done something similar on an Apple Silicon Mac, but I haven't yet re-tried it for this bug. That will follow shortly.

Comment 3

[Steven Michaud [:smichaud] (Retired)]

Note that this behavior is "normal" for all try builds, on any (supported) version of macOS. It happens because these builds aren't signed at all. The workaround is to run xattr -c target.dmg before you double-click on the archive to install its contents.

Comment 4

[Steven Michaud [:smichaud] (Retired)]

(Following up comment #2)

I had the same results on an Apple Silicon Mac running macOS 13 (build 22A380) -- no problems. This machine (actually a partition on it) was initially upgraded from macOS 12 to one of the macOS 13 betas. I continued upgrading the OS through all the subsequent betas, and removed the partition from Apple's beta program when the release (build 22A380) came out.

Comment 5

[:glob ✱]

I had the same problem with an old version of devEdition (installed 9th Sep 2022). Wasn't able to start it due to the same error, even after I removed the signature.

The most recent version works without issue.

Comment 6

[Stephen A Pohl [:spohl]]

It would be great to know what's displayed in Console.app when the build fails to open. Also, codesign -vvvv [path-to-build]/FirefoxNightly.app might display something useful.

Comment 7

[Gian-Carlo Pascutto [:gcp]]

Attachment: codesign.txt

Comment 8

[Gian-Carlo Pascutto [:gcp]]

It would be great to know what's displayed in Console.app when the build fails to open.

Just "zsh: killed". The build is from the 28th of October.

Comment 9

[Stephen A Pohl [:spohl]]

Interesting. What does /Applications/FirefoxNightly.app/Contents/Resources/defaults/pref/channel-prefs.js contain?

Comment 10

[Gian-Carlo Pascutto [:gcp]]

I marked this as S2 because my regular Firefox release install works.

Comment 11

[Stephen A Pohl [:spohl]]

(In reply to Stephen A Pohl [:spohl] from comment #9)

Interesting. What does /Applications/FirefoxNightly.app/Contents/Resources/defaults/pref/channel-prefs.js contain?

I just compared this locally and it looks like updates fail to update the channel-prefs.js file. I'm going to reassign this to app update for someone to take a look. Specifically, we don't seem to be updating these lines that were introduced in bug 1576546:

https://searchfox.org/mozilla-central/rev/3c194fa1d6f339036d2ec9516bd310c6ad612859/browser/app/profile/channel-prefs.js#5-8

Comment 12

[Stephen A Pohl [:spohl]]

Actually, we might be able to verify that this is the culprit. Gian-Carlo, does the build start if you manually replace /Applications/FirefoxNightly.app/Contents/Resources/defaults/pref/channel-prefs.js with the channel-prefs.js file from a fresh download of Nightly? And does codesign -vvvv now show that the app bundle is properly signed?

Comment 13

[Gian-Carlo Pascutto [:gcp]]

Attachment: channel-prefs.js

Comment 14

[Gian-Carlo Pascutto [:gcp]]

And does codesign -vvvv now show that the app bundle is properly signed?

After doing those steps:

I still get the error but codesign -vvvv does report that the app is now valid on disk/statisfies its Designated Requirement.

Comment 15

[Gian-Carlo Pascutto [:gcp]]

Cmd+Click was now enough to open Nightly and update to today's' build. I think your theory is correct.

Comment 16

[Stephen A Pohl [:spohl]]

The one other thing I would have suggested would have been to compare xattr -r for the failing build, and a fresh new download. I can't say if this would have shown any differences, but it could have been another data point.

Comment 17

[[e-mail] (:bhearsum)]

It's possible this is a(In reply to :glob ✱ from comment #5)

I had the same problem with an old version of devEdition (installed 9th Sep 2022). Wasn't able to start it due to the same error, even after I removed the signature.

The most recent version works without issue.

If this is indeed the exact same issue it's plausible that something on the code signing or update generation side is/was at fault. I poked RelEng to see if anything changed there.

Comment 18

[[e-mail] (:bhearsum)]

Another thing that would be useful is an update log right after the faulty update was applied. You can find that through:
 1. Navigate to about:support
 2. Find the "Update Folder" entry and click "Open Folder".
 3. Open the updates directory.
 4. Inside, you should find files named last-update.log and backup-update.log. Attach these files to this bug.

Any chance those are still around, gcp?

Comment 19

[Gian-Carlo Pascutto [:gcp]]

right after the faulty update was applied.

See comment 15, the faulty update is already gone.

Comment 20

[Robin Steuber (she/her) [:bytesized]]

(In reply to Stephen A Pohl [:spohl] from comment #11)

I just compared this locally and it looks like updates fail to update the channel-prefs.js file.

This is by design. I wrote a bit about this in Bug 1760704, where I added a test that specifically tries to prevent anyone from making changes to this file again in order to avoid breaking the signature. But I've never heard of a signature break causing an issue like this. Usually, as long as Firefox has been run before the signature breaks, it continues to work just fine. And since Firefox needs to run in order to update, it hasn't been a problem.

Sadly, there is no good way that I know of for us to fix this bug. We could change the file back, which would fix anyone's installation that was made over 3 years ago. But would cause the exact same issue with newer installations. Which doesn't really seem like an improvement.

Since this happened as a result of an OS upgrade, it makes me wonder if the new version of macOS is verifying signatures of existing apps in way that old OS versions did not.
@spohl - Would you know the answer to this?

Comment 21

[Stephen A Pohl [:spohl]]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #20)

Since this happened as a result of an OS upgrade, it makes me wonder if the new version of macOS is verifying signatures of existing apps in way that old OS versions did not.
@spohl - Would you know the answer to this?

Yes, we have known for a while that Apple would start becoming much more strict about valid signatures. It can be expected that the screws will continue to tighten with every major release of macOS. I vaguely remembered that we did exclude certain files during updates, and it's unfortunate that we got bit here.

Adding a test to ensure that channel-prefs.js never gets changed was a great first step. Could we go one step further and add a test that would actually use the codesign tool and parse the output to ensure that we always have a valid signature, both for fresh installs and updates?

Comment 22

[Robin Steuber (she/her) [:bytesized]]

(In reply to Stephen A Pohl [:spohl] from comment #21)

Could we go one step further and add a test that would actually use the codesign tool and parse the output to ensure that we always have a valid signature, both for fresh installs and updates?

I can foresee some difficulties making this work (in such a way that it would actually be useful). But I filed a bug for this and documented those difficulties: Bug 1799521.

Comment 23

[Julien Cristau [:jcristau]]

We occasionally do need to change channel-prefs.js in updates, e.g. https://archive.mozilla.org/pub/firefox/releases/custom-updates/switch-to-esr78.0-eol-mac.mar from bug 1637533. Does this mean such updates would now need to be complete mars and update app signatures?

Comment 24

[Julien Cristau [:jcristau]]

ni for comment 23

Comment 25

[Johan Lorenzo [:jlorenzo]]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #0)

I see a similar behavior as in bug 1781111 but this isn't mozregression, it's my main browser install.

For the record, bug 1590717 showed a similar behavior too. It was on a release build.

Comment 26

[Stephen A Pohl [:spohl]]

(In reply to Julien Cristau [:jcristau] from comment #23)

We occasionally do need to change channel-prefs.js in updates, e.g. https://archive.mozilla.org/pub/firefox/releases/custom-updates/switch-to-esr78.0-eol-mac.mar from bug 1637533. Does this mean such updates would now need to be complete mars and update app signatures?

I can't say if they need complete mars vs partial as I'm not very familiar with the differences there, but yes, we should absolutely work towards always having valid signatures on our bundles or we risk having severe breakage on future versions of macOS. This ranges from an inability to run Firefox, inability to update Firefox and everything in between, for example being rejected by the macOS Keychain once we have integration with the native password manager.

Comment 27

[Julien Cristau [:jcristau]]

I guess this bug means we need to reconsider how we handle channel-prefs.js and update-settings.ini in updates. Currently they're excluded from update mars, so the file contents are set at initial install time or can be updated by the user (typically for testing on test channels).
Presumably there's a few ways this can break the signature: if the in-tree file contents change (as in bug 1576546), if the user edits the file, or when a beta user updates to a release candidate build.

I can think of two things we can (should?) do:

• short term, do a one-time update for mac users that includes channel-prefs.js so users whose install predates bug 1576546 get a matching signature again
• revisit how we handle the beta / release-candidate stuff. I guess we could create 2 versions of those updates, one with the beta settings and one with the release ones, and give out the right one in balrog? At that point we can stop excluding files from the mars.

That doesn't cover the testing use case. Maybe there should be an override for app.update.channel outside the app bundle?

Does that make sense?

Comment 28

[[e-mail] (:bhearsum)]

(In reply to Julien Cristau [:jcristau] from comment #27)

I guess this bug means we need to reconsider how we handle channel-prefs.js and update-settings.ini in updates. Currently they're excluded from update mars, so the file contents are set at initial install time or can be updated by the user (typically for testing on test channels).

I agree that it's important that we address this. Apple steadily locks things down more, so I imagine it's only a matter of time before this becomes an emergency.

Presumably there's a few ways this can break the signature: if the in-tree file contents change (as in bug 1576546), if the user edits the file, or when a beta user updates to a release candidate build.

I can think of two things we can (should?) do:

• short term, do a one-time update for mac users that includes channel-prefs.js so users whose install predates bug 1576546 get a matching signature again

+1

• revisit how we handle the beta / release-candidate stuff. I guess we could create 2 versions of those updates, one with the beta settings and one with the release ones, and give out the right one in balrog? At that point we can stop excluding files from the mars.

That doesn't cover the testing use case. Maybe there should be an override for app.update.channel outside the app bundle?

This would certainly work, but it's very heavyweight...I believe it means we need additional signing/repackage/repackage-signing steps for each locale, and possibly a new build or type of repackage before all of that to tweak the initial build's channel-prefs.js file? We may also have issues with Balrog. I believe the current data model assumes that Beta and Release get the same MAR (with no easy way to change it). A solvable problem (either by changing the data model or using a differerent Release for Beta), but the bifurcation is not ideal.

Julien and I talked a bit elsewhere, and one idea that came up was to move channel-prefs.js out of the .app bundle and into a per-install location. The idea would be to write this file out once (and only once) at first run, and it would need to be done in a place that all users can write to. (If we have per-user copies of this we may end up with mismatched channels if a second user runs Firefox for the first time after it has been updated to an RC from by different user).

The biggest benefit to this idea is that would fully eliminate the possibility of breaking the signature in this way in the future. It also avoids the need to overhaul and expand the CI pipeline, and solves the "how do test problem" noted above.

The big unknown for me here is whether or not we have any location on Mac that we could write this file to. On Windows, we have %PROGRAMDATA%, but we don't use an equivalent for anything on Mac at the moment.

Kirk, Stephen - do you have any thoughts or ideas on how doable the above is, or other ideas?

Comment 29

[Steven Michaud [:smichaud] (Retired)]

I wonder if it'd be possible to use preference files -- like those in ~/Library/Application Support/Firefox or ~/Library/Preferences. Surely those will never be signed.

Comment 30

[[e-mail] (:bhearsum)]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #29)

I wonder if it'd be possible to use preference files -- like those in ~/Library/Application Support/Firefox or ~/Library/Preferences. Surely those will never be signed.

This might cause issues for multi user cases. Eg:

• User A installs Firefox Beta to /Applications
• User A runs Firefox, which writes out channel-prefs.js with beta in it.
• User A updates Firefox Beta and gets an RC build
• User B runs Firefox, which writes out channel-prefs.js with release in it.

...and now Firefox may bounce back and forth between channels because of the mismatch.

Comment 31

[Steven Michaud [:smichaud] (Retired)]

There are some files in ~/Library/Application Support/Firefox that are all-user (or more precisely "all-profile") -- those in Crash Reports and Pending Pings. So there could be others. There also could, in principle, be another subdirectory of ~/Library/Application Support/Firefox devoted to settings that are "by version", instead of "by user" (or "by profile").

The second possibility, especially, would probably require some serious design changes. But I think these ideas are worth considering, for the medium to long term.

The "bouncing back and forth" might actually be a good thing, if it can be managed properly.

Comment 32

[[e-mail] (:bhearsum)]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #31)

There are some files in ~/Library/Application Support/Firefox that are all-user (or more precisely "all-profile") -- those in Crash Reports and Pending Pings. So there could be others. There also could, in principle, be another subdirectory of ~/Library/Application Support/Firefox devoted to settings that are "by version", instead of "by user" (or "by profile").

The second possibility, especially, would probably require some serious design changes. But I think these ideas are worth considering, for the medium to long term.

The "bouncing back and forth" might actually be a good thing, if it can be managed properly.

I might be misunderstanding something, but ~/ is a particular users home directory, no? If so, how do we know which user's ~/Library we want to place the files in?

Comment 33

[Steven Michaud [:smichaud] (Retired)]

~/ is a particular users home directory, no?

You're right. I'm thinking that, in a multi-user setup, "all profile" and "by version" settings would need to go into every user's ~/Library/Application Support/Firefox directory tree. But I certainly haven't thought this out fully. Some killer objection may turn up that makes my ideas impossible.

Safari has betas. I've never used them before. But it'd be interesting to see how Apple manages these problems. At some point I'll give it a look.

Comment 34

[Steven Michaud [:smichaud] (Retired)]

Does Mozilla support installing into ~/Applications?

If we made that the default, all these problems would go away.

This would probably only make sense for a multi-user setup.

Comment 35

[Robin Steuber (she/her) [:bytesized]]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #31)

The "bouncing back and forth" might actually be a good thing, if it can be managed properly.

I sort of fail to see how this could be a good thing, but if there is something that I'm missing I'd certainly like to discuss it.

---

(In reply to Steven Michaud [:smichaud] (Retired) from comment #34)

Does Mozilla support installing into ~/Applications?

If we made that the default, all these problems would go away.

At first glance, I don't think that this is a good idea. I don't love the idea of a 3-user computer requiring Firefox to take up 3 times the space. And I'm not really sure what we do about existing installations. In general, I don't love the idea of saying that we don't support installing Firefox in the default place that applications reside.

From some quick experimentation, it seems like the default "Applications" links in macOS point directly to /Applications, not to some sort of aggregated directory. Which probably means that some users would install Firefox and then not be able to find it. Which doesn't seem like a good experience.

---

(In reply to [e-mail] (:bhearsum) from comment #28)

(In reply to Julien Cristau [:jcristau] from comment #27)

• revisit how we handle the beta / release-candidate stuff. I guess we could create 2 versions of those updates, one with the beta settings and one with the release ones, and give out the right one in balrog? At that point we can stop excluding files from the mars.

Oh, wait. Are we currently breaking signatures for Release Candidates on Beta? That would make this more urgent than I had realized.

The big unknown for me here is whether or not we have any location on Mac that we could write this file to. On Windows, we have %PROGRAMDATA%, but we don't use an equivalent for anything on Mac at the moment.

Kirk, Stephen - do you have any thoughts or ideas on how doable the above is, or other ideas?

I have an idea but I have no idea how reasonable it is. I believe that you can separately sign frameworks in macOS apps. Perhaps we could separately sign just the channel prefs file, allowing it to be excluded from the signature computation for the rest of the application.

If we prefer the idea of a file outside of the app bundle, perhaps /Users/Shared could be a reasonable location?

Comment 36

[Stephen A Pohl [:spohl]]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #35)

I have an idea but I have no idea how reasonable it is. I believe that you can separately sign frameworks in macOS apps. Perhaps we could separately sign just the channel prefs file, allowing it to be excluded from the signature computation for the rest of the application.

We should test this. When I last worked on this I got the impression that with v2 codesigning, there is no way to exclude anything from "signature computation" anymore. The idea is that nothing inside the bundle is allowed to change or the signature will break. So although frameworks can be signed separately, once they're bundled as part of a .app bundle, it isn't possible to replace it with a different, signed framework of the same name or the app bundle signature will break.

Comment 37

[Steven Michaud [:smichaud] (Retired)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #35)

(In reply to Steven Michaud [:smichaud] (Retired) from comment #31)

The "bouncing back and forth" might actually be a good thing, if it can be managed properly.

I sort of fail to see how this could be a good thing, but if there is something that I'm missing I'd certainly like to discuss it.

I was just thinking out loud. I haven't figured out exactly how this might work.

---

(In reply to Steven Michaud [:smichaud] (Retired) from comment #34)

Does Mozilla support installing into ~/Applications?

If we made that the default, all these problems would go away.

At first glance, I don't think that this is a good idea. I don't love the idea of a 3-user computer requiring Firefox to take up 3 times the space. And I'm not really sure what we do about existing installations. In general, I don't love the idea of saying that we don't support installing Firefox in the default place that applications reside.

From some quick experimentation, it seems like the default "Applications" links in macOS point directly to /Applications, not to some sort of aggregated directory. Which probably means that some users would install Firefox and then not be able to find it. Which doesn't seem like a good experience.

I suspect that, in the not too distant future, Mozilla will need to either support installing to ~/Applications or give up supporting multi-user installations. Yes, it would mean a significant change in the user-facing part of the installation process. Mozilla would probably need to write an installer, and ask whether or not the "host" is a multi-user computer. With luck this won't be needed right away. But given Apple's growing strictness WRT app signing, this is almost certainly where things will end up. Mozilla should start thinking about it before it's about to happen in the next macOS release.

Comment 38

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #28)

(In reply to Julien Cristau [:jcristau] from comment #27)

• revisit how we handle the beta / release-candidate stuff. I guess we could create 2 versions of those updates, one with the beta settings and one with the release ones, and give out the right one in balrog? At that point we can stop excluding files from the mars.

Oh, wait. Are we currently breaking signatures for Release Candidates on Beta? That would make this more urgent than I had realized.

I'm pretty sure we are (jcristau can probably confirm this).

The big unknown for me here is whether or not we have any location on Mac that we could write this file to. On Windows, we have %PROGRAMDATA%, but we don't use an equivalent for anything on Mac at the moment.

Kirk, Stephen - do you have any thoughts or ideas on how doable the above is, or other ideas?

I have an idea but I have no idea how reasonable it is. I believe that you can separately sign frameworks in macOS apps. Perhaps we could separately sign just the channel prefs file, allowing it to be excluded from the signature computation for the rest of the application.

If we prefer the idea of a file outside of the app bundle, perhaps /Users/Shared could be a reasonable location?

That seems plausible, although I wonder about security concerns? Maybe we don't care since you'd need an account on the system to change the file anyways....

I have to admit that this idea really does not appeal to me very much, but I'm hard pressed to come up with anything else that would allow our RC system to work. (The alternative crazy idea is to just stop shipping RCs to Beta on Mac....but that has its own major downsides.)

Comment 39

[Robin Steuber (she/her) [:bytesized]]

(In reply to [e-mail] (:bhearsum) from comment #38)

That seems plausible, although I wonder about security concerns? Maybe we don't care since you'd need an account on the system to change the file anyways....

If we want somewhere more secure, there is /Library/Application Support. But the issue with picking a more secure location will be figuring out how to actually write there from Firefox.

Comment 40

[Julien Cristau [:jcristau]]

(In reply to [e-mail] (:bhearsum) from comment #38)

(In reply to [e-mail] (:bhearsum) from comment #28)

(In reply to Julien Cristau [:jcristau] from comment #27)

• revisit how we handle the beta / release-candidate stuff. I guess we could create 2 versions of those updates, one with the beta settings and one with the release ones, and give out the right one in balrog? At that point we can stop excluding files from the mars.

Oh, wait. Are we currently breaking signatures for Release Candidates on Beta? That would make this more urgent than I had realized.

I'm pretty sure we are (jcristau can probably confirm this).

From the description here it sounds like we are, but I'd be hard pressed to confirm, never having used a mac. :)

Comment 43

[[e-mail] (:bhearsum)]

(In reply to Stephen A Pohl [:spohl] from comment #36)

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #35)

I have an idea but I have no idea how reasonable it is. I believe that you can separately sign frameworks in macOS apps. Perhaps we could separately sign just the channel prefs file, allowing it to be excluded from the signature computation for the rest of the application.

We should test this. When I last worked on this I got the impression that with v2 codesigning, there is no way to exclude anything from "signature computation" anymore. The idea is that nothing inside the bundle is allowed to change or the signature will break. So although frameworks can be signed separately, once they're bundled as part of a .app bundle, it isn't possible to replace it with a different, signed framework of the same name or the app bundle signature will break.

I tried the following:

• Pull the Mozilla VPN.app out of https://archive.mozilla.org/pub/vpn/releases/2.9.0/mac/MozillaVPN.pkg
• Ensure that codesign -vvvv is happy (it was)
• Replace its Contents/Library/LoginItems/MozillaVPNLoginItem.app subbundle with the version from https://archive.mozilla.org/pub/vpn/releases/2.8.0/mac/MozillaVPN.pkg (which I confirmed to be different).

codesign -vvvv on the new outer bundle gave me the exact same output as before I modified it.

On the other hand, if I do something to break the signature of MozillaVPNLoginItem.app, the outer package signature is no longer valid.

All of this was done on an Intel machine running 13.0.1 (I would love to know what the result is on an M1 machine, given that they tend to be more locked down).

Stephen (or anyone else) - was this a valid test of the bolded text above?

Comment 44

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #43)

(In reply to Stephen A Pohl [:spohl] from comment #36)

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #35)

I have an idea but I have no idea how reasonable it is. I believe that you can separately sign frameworks in macOS apps. Perhaps we could separately sign just the channel prefs file, allowing it to be excluded from the signature computation for the rest of the application.

We should test this. When I last worked on this I got the impression that with v2 codesigning, there is no way to exclude anything from "signature computation" anymore. The idea is that nothing inside the bundle is allowed to change or the signature will break. So although frameworks can be signed separately, once they're bundled as part of a .app bundle, it isn't possible to replace it with a different, signed framework of the same name or the app bundle signature will break.

I tried the following:

• Pull the Mozilla VPN.app out of https://archive.mozilla.org/pub/vpn/releases/2.9.0/mac/MozillaVPN.pkg
• Ensure that codesign -vvvv is happy (it was)
• Replace its Contents/Library/LoginItems/MozillaVPNLoginItem.app subbundle with the version from https://archive.mozilla.org/pub/vpn/releases/2.8.0/mac/MozillaVPN.pkg (which I confirmed to be different).

codesign -vvvv on the new outer bundle gave me the exact same output as before I modified it.

On the other hand, if I do something to break the signature of MozillaVPNLoginItem.app, the outer package signature is no longer valid.

All of this was done on an Intel machine running 13.0.1 (I would love to know what the result is on an M1 machine, given that they tend to be more locked down).

Stephen (or anyone else) - was this a valid test of the bolded text above?

I did another test, this time with Firefox. I:

• Downloaded Nightly and Release
• Replaced Nightly's crashreporter.app with Release's
• Verified that the signature was still good, and that Firefox ran.

The only piece that this doesn't test is that Firefox would still work after an update happens that doesn't touch the embedded app (I don't have an easy way to, eg: exclude crashreporter.app from an update to simulate that) -- but this seems like a very strong sign that we can modify a bundled app without breaking the outer signature.

I also did some research on this to try to find some supporting documenation. I found a couple of relevant notes on https://developer.apple.com/library/archive/technotes/tn2206/_index.html#//apple_ref/doc/uid/DTS40007919-CH1-TNTAG402:

Changes That Don't Invalidate a Code Signature

There are a few changes you can make to a signed bundle that won't invalidate its signature.

If you have optional or replaceable content you wish to change without invalidating the code signature, nested code can be replaced with equivalent (conforms to the designated requirement) nested code without disturbing the outer signature. This is the design mechanism for indirection in code bundles. It is acceptable to code-sign a bundle containing no main executable, and then treat it as nested code (typically in Contents).

I think this bit is confirming my test?

There's also a note that says:

Conversely, putting code into other places will cause it to be sealed as data (resource) files. This causes this code to be sealed twice; once in the nested code, and once in the outer signature. This wastes both signing and verification time and storage space. Also, this can break the outer signature of apps that use their own update mechanisms to replace nested code. If this nested code is being treated as resources, the outer signature doesn't know that this nested content is actually code.

...which seems to imply that whether it's what we're already talking about, or something different, that it should be possible for us to keep channel-prefs.js in some sort of sub-bundle.

Comment 45

[Nick Alexander :nalexander [he/him]]

...which seems to imply that whether it's what we're already talking about, or something different, that it should be possible for us to keep channel-prefs.js in some sort of sub-bundle.

I'll add that such a mechanism might also allow us to finally include attribution data in a robust manner.

Comment 46

[Robin Steuber (she/her) [:bytesized]]

(In reply to Nick Alexander :nalexander [he/him] from comment #45)

...which seems to imply that whether it's what we're already talking about, or something different, that it should be possible for us to keep channel-prefs.js in some sort of sub-bundle.

I'll add that such a mechanism might also allow us to finally include attribution data in a robust manner.

I think we would have to sign the attribution data for that to work. I don't know for sure if that's a sticking point or not, but I don't think we've ever done something like that before.

Comment 47

[[e-mail] (:bhearsum)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #46)

(In reply to Nick Alexander :nalexander [he/him] from comment #45)

...which seems to imply that whether it's what we're already talking about, or something different, that it should be possible for us to keep channel-prefs.js in some sort of sub-bundle.

I'll add that such a mechanism might also allow us to finally include attribution data in a robust manner.

I think we would have to sign the attribution data for that to work. I don't know for sure if that's a sticking point or not, but I don't think we've ever done something like that before.

If I've understood this all correctly, the only things that can be changed without breaking the outer package signature, are things that are indepedently signed (namely: subbundles). As far as I know, all non-code files must go in Resources, and cannot be modified without breaking the signature.

Comment 48

[Stephen A Pohl [:spohl]]

(In reply to [e-mail] (:bhearsum) from comment #44)

[...]
...which seems to imply that whether it's what we're already talking about, or something different, that it should be possible for us to keep channel-prefs.js in some sort of sub-bundle.

This sounds very promising indeed!

Comment 49

[Robin Steuber (she/her) [:bytesized]]

Ok, so it sounds to me like the best path forward is individually signing channel-prefs.js.

Since channel-prefs.js isn't expected change ever, really, I don't really feel like setting things up to have it signed when we sign the rest of the app bundle really feels necessary. To me, it seems like it would be easiest, both in terms of process and maybe also for testing, to just sign the thing once (per channel) and then check the signed sub-bundles into the tree and (on macOS) include the correct one in the install instead of channel-prefs.js.

Since we will be continuing to not update it under normal circumstances, we will probably want to do a one-time update of our macOS users to get everyone to a good state where they have the signed sub bundle and not the unsigned version so that everyone ends up with their signature validating. I'm not sure exactly who the right person to talk to about this is.

@bhearsum - Does this sound reasonable to you? I'm a bit unsure about the "once per channel" bit. Would that actually only be necessary for 5 channels, or would that end up exploding into way more than I am thinking for funny suffixed channels that I am only kind of aware of the existence of? Do you know what I need to do to get the ball rolling for the "one-time update" bit?

Comment 50

[Robin Steuber (she/her) [:bytesized]]

I'm not sure how much of the work here I am actually going to end up doing myself, but I'm going to assign this to myself since I'm driving it. I'll file blocking bugs for work that ought to be done by others, when that becomes appropriate.

Comment 51

[[e-mail] (:bhearsum)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #49)

Ok, so it sounds to me like the best path forward is individually signing channel-prefs.js.

Since channel-prefs.js isn't expected change ever, really, I don't really feel like setting things up to have it signed when we sign the rest of the app bundle really feels necessary. To me, it seems like it would be easiest, both in terms of process and maybe also for testing, to just sign the thing once (per channel) and then check the signed sub-bundles into the tree and (on macOS) include the correct one in the install instead of channel-prefs.js.

This might be OK -- the downside is that anytime anything changes about it, or we need to add a new channel - we'd have to do manual work to create the new bundle. This is a fairly rare event, but I'm a little bit hesitant to introduce new manual steps that might block releases. It would also mean that if we ever need to test something on some non-real channel (which we do at times), we'd need to deal with bundle generation first. I think I would prefer that we put the bundle generation in the tree at the very least, although I'd be totally fine with it being dealt with as a toolchain or otherwise as an upstream dependency of Firefox (to avoid building it over and over and over).

I'm happy for RelEng to take on all of the build system & automation bits if it helps push us make this fully automated.

Since we will be continuing to not update it under normal circumstances, we will probably want to do a one-time update of our macOS users to get everyone to a good state where they have the signed sub bundle and not the unsigned version so that everyone ends up with their signature validating. I'm not sure exactly who the right person to talk to about this is.

I discussed this a bit with jcristau, and I think what we need to do is include the new bundle files with add-if-not instructions, and remove channel-prefs.js at the same time. (We'll also need a watershed on beta to make sure that those users get the beta bundle, and not the release bundle.)

Is that what you're talking about, or something else?

@bhearsum - Does this sound reasonable to you? I'm a bit unsure about the "once per channel" bit. Would that actually only be necessary for 5 channels, or would that end up exploding into way more than I am thinking for funny suffixed channels that I am only kind of aware of the existence of?

I think there's a few categories of things here:

• Distribution builds, which add suffixes at runtime - so those aren't a concern here.
• Non-official channels (eg: pine, oak, where we sometimes do development that requires updates). These would need their own bundles.
• Test channels that QA uses to verify updates. These would also either need their own bundles, or a way to set or suffix the channel at runtime. Manually signed bundles that they can drop in are probably OK for this case. (We can't use the distribution mechanism to suffix, because it will always add -cck- before the distribution id - which doesn't match our test channel names.)

Comment 52

[Robin Steuber (she/her) [:bytesized]]

(In reply to [e-mail] (:bhearsum) from comment #51)

I think I would prefer that we put the bundle generation in the tree at the very least, although I'd be totally fine with it being dealt with as a toolchain or otherwise as an upstream dependency of Firefox (to avoid building it over and over and over).\

Yeah, you've convinced me that this is the way to go.

I'm happy for RelEng to take on all of the build system & automation bits if it helps push us make this fully automated.

That would be great. Do you need anything to make that happen? I'm slightly unclear on what the work looks like there.

Comment 53

[[e-mail] (:bhearsum)]

(In reply to Kirk Steuber (he/him) [:bytesized] from comment #52)

(In reply to [e-mail] (:bhearsum) from comment #51)

I think I would prefer that we put the bundle generation in the tree at the very least, although I'd be totally fine with it being dealt with as a toolchain or otherwise as an upstream dependency of Firefox (to avoid building it over and over and over).\

Yeah, you've convinced me that this is the way to go.

I'm happy for RelEng to take on all of the build system & automation bits if it helps push us make this fully automated.

That would be great. Do you need anything to make that happen? I'm slightly unclear on what the work looks like there.

I see a few parts here:

1. Something (probably build system goop) needs to package channel-prefs.js into a bundle
2. Adjustments to any code that currently depends on channel-prefs.js to cope with the new location
3. Ensure new bundle will be signed (I think this will happen automatically, but we ought to confirm that)
4. Set-up the watershed for Beta

#3 & #4 are obviously RelEng tasks, and I think you are probably best positioned to deal with #2. #1 - I'm not really sure. Stephen - would you be willing to take that part on? I'm sure Kirk or I could probably muddle our way through it, but it feels you it's probably up your alley more (correct me if I'm wrong though).

Comment 54

[Stephen A Pohl [:spohl]]

Yes, I should be able to take a look at this soon-ish. Keeping n-i set.

Comment 55

[[e-mail] (:bhearsum)]

Attachment: WIP: Bug 1799332: Stop including channel-prefs.js and update-settings in macOS packages

Note: this is absolutely not landable on its own. It should have everything that is needed to remove these files from the packages and updates, but there's tons of app and test code that needs updating before this is landable.

I verified that the removals do work in a try push. Partial MARs end up with remove instructions, and the complete MARs simply do not include the files at all. (removed-files.in suggests that the updater will handle this automatically, but it would be good to get some verification of that...)

Comment 56

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Switch to using xattr instead of channel-prefs.js in most instances on macOS, with the exception of channel switching (typically to move users to ESR). r=bhearsum,bytesized

Comment 57

[Nick Alexander :nalexander [he/him]]

spohl: is your approach to set an extended attribute on the .app bundle? How does that get put in place for distribution? I'm pursuing the same idea for a different use case, and I've got some WIP to our Linux tooling to support extended attributes correctly that might help with this.

But now I see the attachment marked as obsolete. Can I get some context?

Comment 58

[Stephen A Pohl [:spohl]]

(In reply to Nick Alexander :nalexander [he/him] from comment #57)

spohl: is your approach to set an extended attribute on the .app bundle? How does that get put in place for distribution? I'm pursuing the same idea for a different use case, and I've got some WIP to our Linux tooling to support extended attributes correctly that might help with this.

But now I see the attachment marked as obsolete. Can I get some context?

I marked the patch as obsolete because this was accidentally submitted to phabricator when I was trying to post a different patch for bug 1815473. Yes, I intend to set extended attributes for channel-prefs.js and update-settings.ini on the .app. Can you walk me through the "distribution" use case? For the use cases that I'm dealing with, Firefox itself will be setting these attributes since we don't have (and don't want) an installer to fall back on.

Comment 59

[Stephen A Pohl [:spohl]]

Attachment: WIP: Bug 1799332: Switch to using xattr instead of channel-prefs.js in most instances on macOS, with the exception of channel switching (typically to move users to ESR).

Comment 60

[Robin Steuber (she/her) [:bytesized]]

Hey, I just wanted to check in on what the status of this is. Are you still working on this? Are there any blockers here?

Comment 61

[Stephen A Pohl [:spohl]]

No blockers and I'm working on it. But I was OOO all week last week and have been getting pulled in a few different directions. This is next on the list.

Comment 62

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange

Comment 63

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Stop bundling channel-prefs.js on macOS. r=bytesized

Comment 64

[Mike Kaply [:mkaply]]

Will we have the ability to change the Framework via an update similar to what we did to move older macOS folks over to the ESR? We did this by updating channel-prefs.js only.

Comment 65

[Stephen A Pohl [:spohl]]

(In reply to Mike Kaply [:mkaply] from comment #64)

Will we have the ability to change the Framework via an update similar to what we did to move older macOS folks over to the ESR? We did this by updating channel-prefs.js only.

Yes, that's precisely why the Framework approach was chosen. As long as the Framework is signed, it can be replaced in the application bundle and will now no longer invalidate the signature on the .app. This will allow us to move folks to ESR, keep beta users on the beta channel while testing RC builds, allow QA to test updates by replacing the Framework instead of channel-prefs.js etc.

Comment 66

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized

Comment 67

[BMO Github Automation]

Attachment: [mozilla-releng/scriptworker-scripts] Bug 1799332 - Add support for signing gecko frameworks in iscript (#838)

Comment 68

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized

Depends on D186139

Comment 69

[Stephen A Pohl [:spohl]]

I'm going to rewrite the README.md for the ChannelPrefs framework, but figured I'd post this here to help in the future if anyone is interested in understanding the reasons behind the change away from channel-prefs.js to using a Framework:

ChannelPrefs macOS Framework

Summary

The ChannelPrefs macOS Framework is used to initialize the app.update.channel
pref during startup.

What is this Framework replacing?

In the past, the app.update.channel pref was initialized by reading the pref
from a file called channel-prefs.js. This file used to be bundled in the .app
bundle under Contents/Resources.

What is app.update.channel and what is it used for?

app.update.channel is used to set the download channel for application
updates.

Why was a separate file (channel-prefs.js) ever necessary to set this pref?

There are three main use cases that made it necessary for the
app.update.channel pref to be set by an external file. These are described in
more detail below:

1. Allowing users on the Beta channel to test RC builds

Beta users have their update channel set to 'beta'. However, RC builds by
definition have their channel set to release, since these are release
candidates that could get released to our release population. If we were to
indiscriminately update our Beta users to an RC build, we would lose our entire
Beta population since everyone would get switched to the release channel.

2. Switching users to another channel, such as ESR

In contrast to the Beta use case outlined above, there are times where we
explicitly WANT to switch users to a different channel. An example of this is
when hardware or a particular macOS version has reached their EOL. In this case,
we usually switch users to our ESR channel for extended support.

3. QA update testing

QA requires a way to temporarily switch the update channel to a test channel in
order to test updates before new releases.

How was channel-prefs.js used to address these use cases?

We were able to accommodate all three use cases above by enabling the updater to
ignore certain files on disk if they were already present, but continue to force
update them if so desired.

In the case of a Beta user updating to an
RC build, the updater would encounter a channel-prefs.js file inside the .app
bundle that has an update channel of beta. In this case, the updater would
not update channel-prefs.js, but update everything else. This beta user is now
able to test the RC build with the update channel still set to beta.

In the case of switching users to the ESR channel, the updater would be set to
forcefully update channel-prefs.js, even if already present on disk. After the
update, the user would now be set to the esr channel and start receiving
updates through this channel.

Before releases, QA would replace the channel-prefs.js file within the .app
bundle and point the channel at a test channel in order to test updates. During
testing, the new channel-prefs.js file would remain in place for typical
update testing, but would get replaced in case QA was testing channel switching.

Why could we not continue to use channel-prefs.js?

Apple has started to strengthen code signature checks and the requirements on
developers such as ourselves on how their apps are signed. In particular,
the channel-prefs.js under Contents/Resources is now included in signature
verifications. Although this was not an issue in the common case, there were two
particular cases that have started to cause problems:

1. Beta users on RC builds

Beta users who were actively running an RC build were technically running a
.app bundle with an invalid signature. This is due to the fact that RC builds
are packaged and signed with a channel-prefs.js file with the
app.update.channel set to release. Since our updater did not update the
channel-prefs.js file since Beta users already have one on disk with the
channel set to beta, an explicit signature check on the .app bundle on disk
would yield an "invalid signature" result. Although macOS does not commonly
re-verify application signatures after they were verified a first time after
install, Apple may start doing so in the future. One instance where such an
additional signature check already occurs is after a macOS version upgrade: If
a user upgrades from macOS 12 to macOS 13 for example, macOS will re-verify the
code signature on the .app bundle. This can cause problem for example if a Beta
user is actively testing an RC build due to the user having a channel-prefs.js
file in the .app bundle that is set to the "beta" channel, rather than "release"
as was the case when the RC build was codesigned.

2. Developers accidentally changing the contents of channel-prefs.js

There have been a few instances where channel-prefs.js was accidentally
modified without the necessary precautions. One such instance was a simple
update to the copyright header in the file. Although seemingly innocuous, this
simple change caused users to have invalid code signatures on their .app bundles
since the updater wouldn't update the channel-prefs.js file on a user's disk if
one was already present. This was even more impactful than Beta users
temporarily testing RC builds since these users never returned to a valid code
signature.

Why is a macOS Framework the best solution to replace channel-prefs.js?

A macOS Framework is the ideal solution to replacing channel-prefs.js because
Frameworks are the only component within a .app bundle that can be replaced
without invalidating the code signature on the .app bundle, as long as both the
previous and the new Framework are signed. This addresses all shortcomings of
channel-prefs.js outlined above, while maintaining the full set of functionality
that channel-prefs.js provided.

Comment 70

[Stephen A Pohl [:spohl]]

Attachment: Bug 1799332: Enable signing of ChannelPrefs.framework. r=haik

Depends on D186363

Comment 71

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/7ae076c476ea
Stop bundling channel-prefs.js on macOS. r=bytesized

Comment 72

[Robin Steuber (she/her) [:bytesized]]

Nope! Not quite, Pulsebot. That was one changeset pushed to Pine only.

Comment 73

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/136277f3604f
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/530823230778
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/b16401e56e06
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/a78e6f0e17ea
Enable signing of ChannelPrefs.framework. r=haik

Comment 74

[Robin Steuber (she/her) [:bytesized]]

Gah, same issue as before. I need to find out if there is a way to turn off this behavior

Comment 75

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/385cfca8c47a
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/51cfafed5a76
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/67b60c6c061f
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/e06019eac6f3
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/1fe3c8cb7f8b
Enable signing of ChannelPrefs.framework. r=haik

Comment 76

[Florian Quèze [:florian]]

(In reply to Robin Steuber (they/them) [:bytesized] from comment #74)

Gah, same issue as before. I need to find out if there is a way to turn off this behavior

Add the leave-open keyword on the bug before landing changes that aren't meant to close the bug as fixed.

Comment 77

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/35a8291df4ba
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/0962b63034ee
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/7cb1a5242461
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/29502456f063
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/3c31ad7040b3
Enable signing of ChannelPrefs.framework. r=haik

Comment 78

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/50a418206fc4
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/f81259f99339
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/a72eb4bf4c5d
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/50abc87d29c0
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/ae9da0a49077
Enable signing of ChannelPrefs.framework. r=haik

Comment 79

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/e4124e150f5a
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/a9e65a01a0ab
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/2fa9cfbafa62
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/a3d8710db904
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/3729cb4a0cc6
Enable signing of ChannelPrefs.framework. r=haik

Comment 80

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/a3b13857feca
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/2149dc5e366c
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/25a54179dcaa
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/49fbfabbf58e
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/b78065ae49bb
Enable signing of ChannelPrefs.framework. r=haik

Comment 81

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/projects/pine/rev/dce1f5cea34d
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/projects/pine/rev/37505796c487
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange
https://hg.mozilla.org/projects/pine/rev/44488eeb6b6b
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/projects/pine/rev/0f1a585d6b6b
Add documentation for ChannelPrefs macOS Framework. r=mstange,bytesized
https://hg.mozilla.org/projects/pine/rev/426aa1316f35
Enable signing of ChannelPrefs.framework. r=haik

Comment 82

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a50eb05b0b0b
Stop bundling channel-prefs.js on macOS. r=bytesized
https://hg.mozilla.org/integration/autoland/rev/d8dd28c68304
Add a macOS Framework to replace the use of channel-prefs.js on macOS. r=glandium,mstange,KrisWright,bytesized
https://hg.mozilla.org/integration/autoland/rev/c7aa9b74be01
Update tests due to no longer bundling channel-prefs.js on macOS and locking the app.update.channel pref. r=bytesized
https://hg.mozilla.org/integration/autoland/rev/62d28922bd23
Add documentation for ChannelPrefs macOS Framework. r=mstange
https://hg.mozilla.org/integration/autoland/rev/1e094e3ca992
Enable signing of ChannelPrefs.framework. r=haik

Comment 83

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/a50eb05b0b0b
https://hg.mozilla.org/mozilla-central/rev/d8dd28c68304
https://hg.mozilla.org/mozilla-central/rev/c7aa9b74be01
https://hg.mozilla.org/mozilla-central/rev/62d28922bd23
https://hg.mozilla.org/mozilla-central/rev/1e094e3ca992