1882729 - rescue mac nightly users affected by bug 1882322

Status: RESOLVED FIXED

(Release Engineering :: Release Automation: Updates, task, P1)

Description

[Julien Cristau [:jcristau]]

Builds affected by bug 1882322 can't apply a complete mar without busting the install.
We can still rescue users who haven't had a complete update yet, by making sure they can get a partial update with the fix.

Comment 1

[Julien Cristau [:jcristau]]

Attachment: create mac-partials balrog blobs

Comment 2

[Julien Cristau [:jcristau]]

We've added new rules to nightly* to ensure that users on mac with build IDs between 20240222100216 (inclusive) and 20240229093832 (exclusive) get only partial updates. Once on 20240229093832 things should work properly again.

Users who did get a complete update while in the affected range can't update further and will need to reinstall, there's likely nothing we can do to salvage that.

Comment 3

[Geoff Brown [:gbrown] (pto, back Jun 10)]

The new rules seemed to bust the nightly-desktop decision task: https://treeherder.mozilla.org/logviewer?job_id=449005873&repo=mozilla-central

The new rules/releases were deleted and the nightly decision task is running again.

Comment 4

[Julien Cristau [:jcristau]]

Attachment: create mac-partials balrog blobs

Updated to add new buildids, and changed the naming scheme to not break the nightly cron decision task's assumptions.

Comment 5

[Julien Cristau [:jcristau]]

Attachment: create mac-partials balrog blobs

Updated to change URLs. Files under https://archive.mozilla.org/pub/firefox/nightly/partials expire after 12 days, so we're planning to copy these to the corresponding directory under https://archive.mozilla.org/pub/firefox/nightly/2024/ to avoid having to worry about necessary partials going away next week.

Comment 6

[Julien Cristau [:jcristau]]

Attachment: copy partials to permanent storage

Comment 7

[[e-mail] (:bhearsum)]

Comment on attachment 9388679 [details]
copy partials to permanent storage

Just for posterity:

• The reason we don't need to grep away 2024-02-0 is because those builds were all 124.0a1
• We also don't need to worry about 2024-03 at all because we were lucky enough to fix this just in time for the first 2024-03 nightly.

Comment 8

[Julien Cristau [:jcristau]]

Attachment: gsutil-commands

Comment 9

[Julien Cristau [:jcristau]]

Attachment: gsutil-commands

Turns out running a handful of gsutil commands and telling them to copy in parallel is more efficient than running 4k+ commands sequentially.

Comment 10

[Julien Cristau [:jcristau]]

Comment on attachment 9388679 [details]
copy partials to permanent storage

replacement:

for src in $(gsutil ls -d gs://moz-fx-productdelivery-pr-38b5-productdelivery/pub/firefox/nightly/partials/2024/0*/2024-0*-mozilla-central* | 
    grep -v 2024-02-1 |
    grep -v 2024-02-2[0123]); do
dst=$(echo $src | sed 's,partials/,,;s,$,partials/,')
echo gsutil -m cp ${src}firefox-mozilla-central-125.0a1-mac-*.partial.mar ${dst}
done

Comment 11

[Julien Cristau [:jcristau]]

The new update rules were added on Friday, and updates re-enabled.