1883123 - make release automation work with new macOS update frameworks

Status: RESOLVED FIXED

(Release Engineering :: Release Automation, defect)

Description

[[e-mail] (:bhearsum)]

https://bugzilla.mozilla.org/show_bug.cgi?id=1799332 got rid of update-settings.ini and channel-prefs.js and replaced them with binary Frameworks. This will likely break update verify for two reasons:

1. We run mac update verify on Linux, and need an update-settings.ini for it to function. (See this function, where we try to copy it into the right place. For this, the best idea I can come up with is pulling from the Linux package the updater binary comes from (to be honest...we probably should've done this from the start).
2. There are a few places where we allow for some differences in channel-prefs.js and update-settings.ini. Because these simple plain text files have been replaced with signed Frameworks, we will most likely need to blanket allow for any differences in the frameworks in any circumstance we expect it.

Comment 1

[Julien Cristau [:jcristau]]

[Tracking Requested - why for this release]: this is a blocker for 125 beta.

Ryan fyi

Comment 2

[[e-mail] (:bhearsum)]

(In reply to [e-mail] (:bhearsum) from comment #0)

2. There are a few places where we allow for some differences in channel-prefs.js and update-settings.ini. Because these simple plain text files have been replaced with signed Frameworks, we will most likely need to blanket allow for any differences in the frameworks in any circumstance we expect it.

This part can probably be deferred a little bit if needed, as it only affects RCs. I may spin it off into a separate bug depending on whether or not I can get it done before 125 uplifts to Beta.

Comment 3

[[e-mail] (:bhearsum)]

I managed to get a simulated sequence of 125.0 betas on Try. The resulting update verify tests fail with:

[task 2024-03-05T21:18:05.048Z] + /builds/worker/checkouts/gecko/tools/update-verify/release/updates/updater/firefox/updater /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app 0
[task 2024-03-05T21:18:05.061Z] Unable to init server: Could not connect: Connection refused
[task 2024-03-05T21:18:05.414Z] + set +x
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: sUsingService=false
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: sUpdateSilently=false
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: gIsElevated=false
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: Writing status to file: applying
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: PATCH DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/update
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: INSTALLATION DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: WORKING DIRECTORY /builds/worker/checkouts/gecko/tools/update-verify/release/updates/source/Firefox.app
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: failed: 38
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: Writing status to file: failed: 38
[task 2024-03-05T21:18:05.415Z] 
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: calling QuitProgressUI
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: Running LaunchCallbackAndPostProcessApps
[task 2024-03-05T21:18:05.415Z] 2024-03-05 21:18:05+0000: No callback arg. Skipping LaunchWinPostProcess and LaunchCallbackApp

...which bytesized tells me is indicative of update-settings.ini not being present.

Comment 4

[[e-mail] (:bhearsum)]

Attachment: Bug 1883123: fully ignore changes in UpdateSettings and ChannelPrefs frameworks for update verify tests r?jcristau!

These new frameworks unavoidably change in unpredictable ways with each build (signing them generate CodeResources, and modifies the included binary). Because of this we have no choice but to simply ignore differences in them when running update verify.

This generally should be OK, because the happy path of updates is one where the on-disk version of these files already exist, and would not be altered as part of updating. The main risk comes in around the first update to include them, where they will be added to existing installs. Ideally we would probably want additional tests around this that remove these frameworks before applying the updates, which we would not want to ignore these files for -- but I doubt we can get that ready in time for 125 hitting Beta.

Comment 5

[[e-mail] (:bhearsum)]

Attachment: Bug 1883123: add support for using host platform update-settings.ini in update verify r?jcristau!

This allows macOS update verify to continue working when update-settings.ini no longer exists in these packages.

Depends on D203801

Comment 6

[[e-mail] (:bhearsum)]

I managed to verify my fixes here on Try, by:

• Creating a 125.0b3 staging release: https://treeherder.mozilla.org/jobs?repo=try&revision=3741c545392a115cf72f90855c14964361d1b207
• Creating a 125.0b4 staging release with no fixes applied, which showed the first of the errors we'd hit: https://treeherder.mozilla.org/jobs?repo=try&revision=f66c3d2536405c479c6126b41eb9999ca26f2abf
• Creating a 125.0b4 staging release with fixes applied: https://treeherder.mozilla.org/jobs?repo=try&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception%2Cusercancel%2Crunning%2Cpending%2Crunnable&revision=dd67bd3fa0db3054a20db4463526606878575a09

Comment 7

[[e-mail] (:bhearsum)]

The patches ended up with some minor updates. I did a new staging run to verify them: https://treeherder.mozilla.org/jobs?repo=try&revision=103b47eb76d9216a3f43f0bf9a2bb4bd473b8b5e

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/49cc33ed1599
fully ignore changes in UpdateSettings and ChannelPrefs frameworks for update verify tests r=jcristau
https://hg.mozilla.org/integration/autoland/rev/d88ec0d14b41
add support for using host platform update-settings.ini in update verify r=jcristau

Comment 9

[Cristina Horotan [:chorotan]]

https://hg.mozilla.org/mozilla-central/rev/49cc33ed1599
https://hg.mozilla.org/mozilla-central/rev/d88ec0d14b41