Bugzilla

Alright, I believe I have figured out the next steps that ought to be taken here:

The unexpectedly-present `channel-prefs.js` issue seems to have been identified and spun off into Bug 1882619. But the worst damage that problem should be able to do is to break the app bundle's signature, which doesn't typically cause problems and can be fixed via update. So I'm not worrying too much about that. It'll merge whenever it merges. Once it does, I'll make sure that things look as they should and the signatures validate.

I believe that the patch I posted to this bug should address the missing framework issue. I plan to land the patch as soon as possible.

Once it has landed, there are a couple things to do before we turn macOS Nightly updates back on. One of them is that I would like to try to run a bit more manual testing by downloading the generated MARs and making sure that applying them works as expected. This should be quick. The remaining things have to do with trying to break as few Nightly installations as possible.

Unfortunately, anyone who is already missing their `UpdateSettings` framework binary is simply not going to be able to update automatically. The [relevant code](https://searchfox.org/mozilla-central/rev/9cd4ea81e27db6b767f1d9bbbcf47da238dd64fa/toolkit/mozapps/update/updater/updater.cpp#2705-2720) simply will not install an update if it can't verify that it is for the correct channel which it cannot do without that framework. Those users will see "Update Failed" notifications that will prompt them to update manually by reinstalling. However, there are other users that are in a bad state but can be "rescued".

The mechanism of this bug is that the frameworks are removed due to erroneous removal instructions in the `precomplete` file. When applying an update from a "complete" MAR, the `precomplete` file is used to find and remove almost all existing installation files. This should not include files that define the current channel, but currently it does. Meaning that the inability to update isn't encountered until after one update writes those files into `precomplete` and then another update, specifically from a complete MAR, reads that `precomplete` file that tells it to delete the frameworks. Thus, we want to be very careful about how we update users that may have a bad `precomplete` file. They should be able to update successfully as long as we are careful to serve them a partial MAR to update them to a good version.

I've talked to :jcristau and :bhearsum about this, and it should be possible to do this via some carefully crafted Balrog rules and likely some manually generated partial MARs. This is the other thing that needs to be done between this patch landing and macOS Nightly updates being re-enabled.

---

To summarize, this is my plan:

1. Merge the attached patch.
2. Run some more quick manual testing on the resulting MARs.
3. Manually generate the necessary partial MARs.
4. Add necessary Balrog rules.
5. Turn macOS Nightly updates back on.

Alright, I believe I have figured out the next steps that ought to be taken here:

The unexpectedly-present `channel-prefs.js` issue seems to have been identified and spun off into Bug 1882619. But the worst damage that problem should be able to do is to break the app bundle's signature, which doesn't typically cause problems and can be fixed via update. So I'm not worrying too much about that. It'll merge whenever it merges. Once it does, I'll make sure that things look as they should and the signatures validate.

I believe that the patch I posted to this bug should address the missing framework issue. I plan to land the patch as soon as possible.

Once it has landed, there are a couple things to do before we turn macOS Nightly updates back on. One of them is that I would like to try to run a bit more manual testing by downloading the generated MARs and making sure that applying them works as expected. This should be quick. The remaining things have to do with trying to break as few Nightly installations as possible.

Unfortunately, anyone who is already missing their `UpdateSettings` framework binary is simply not going to be able to update automatically. The [relevant code](https://searchfox.org/mozilla-central/rev/9cd4ea81e27db6b767f1d9bbbcf47da238dd64fa/toolkit/mozapps/update/updater/updater.cpp#2705-2720) just will not install an update if it can't verify that it is for the correct channel which it cannot do without that framework. Those users will see "Update Failed" notifications that will prompt them to update manually by reinstalling. However, there are other users that are in a bad state but can be "rescued".

The mechanism of this bug is that the frameworks are removed due to erroneous removal instructions in the `precomplete` file. When applying an update from a "complete" MAR, the `precomplete` file is used to find and remove almost all existing installation files. This should not include files that define the current channel, but currently it does. Meaning that the inability to update isn't encountered until after one update writes those files into `precomplete` and then another update, specifically from a complete MAR, reads that `precomplete` file that tells it to delete the frameworks. Thus, we want to be very careful about how we update users that may have a bad `precomplete` file. They should be able to update successfully as long as we are careful to serve them a partial MAR to update them to a good version.

I've talked to :jcristau and :bhearsum about this, and it should be possible to do this via some carefully crafted Balrog rules and likely some manually generated partial MARs. This is the other thing that needs to be done between this patch landing and macOS Nightly updates being re-enabled.

---

To summarize, this is my plan:

1. Merge the attached patch.
2. Run some more quick manual testing on the resulting MARs.
3. Manually generate the necessary partial MARs.
4. Add necessary Balrog rules.
5. Turn macOS Nightly updates back on.