1816992 - Produce "attributable" macOS DMGs with placeholder attribution details in automation

Status: RESOLVED FIXED

(Firefox :: Installer, enhancement, P3)

Description

[Nick Alexander :nalexander [he/him]]

This ticket tracks using the functionality added in Bug 1816982 and Bug 1816986 to set an extended attribute on the main Firefox.app bundle containing placeholder attribution details, and then producing an "attributable" macOS DMG with that extended attribution as part of the release pipeline.

This is sibling to https://bugzilla.mozilla.org/show_bug.cgi?id=1814727, which tracks adding placeholder attribution to Windows installers.

Comment 1

[Nick Alexander :nalexander [he/him]]

It would be nice to always set the xattr when producing DMGs, even just for local builds. My guess is that the "attributable" macOS DMG would be part of the macOS repackage job.

Comment 2

[[e-mail] (:bhearsum)]

(In reply to Nick Alexander :nalexander [he/him] from comment #1)

It would be nice to always set the xattr when producing DMGs, even just for local builds. My guess is that the "attributable" macOS DMG would be part of the macOS repackage job.

Indeed! There's a few things that will need to happen here:

• Update to a newer libdmg-hfsplus after https://github.com/mozilla/libdmg-hfsplus/pull/6 lands
• Produce attributable DMGs as part of repackage - these dmg.py changes will probably be a decent starting point for this, but no doubt it will need some tweaks. At the very least to make attributable build support conditional, and only enable it for certain builds (most likely shippable and devedition). We also need to switch the tag to something consistent across tasks (most likely __MOZCUSTOM__, since it's what we already use for this purpose on Windows). Other tweaks will likely be needed too.
• Create attribution tasks for macOS that will use dmg attribute to replace the tag with sentinel attribution information. Most likely this will be the same as what we use for windows, but I still need to double check with a few stakeholders to confirm that. The implementation will probably end up looking similar to the sentinel attribution implementation for Windows.
• Update beetomver tasks to pull DMGs from the new attribution tasks instead of repackage (pkgs will still need to come from repackage, however).

Comment 3

[Heitor Neiva [:hneiva]]

Attachment: WIP: Bug 1816992 - Add sentinel attribution to repackage jobs r?bhearsum

Comment 4

[Heitor Neiva [:hneiva]]

Attachment: WIP: Bug 1816992 - Add attribution job for DMGs r?bhearsum

Depends on D188855

Comment 5

[Heitor Neiva [:hneiva]]

Attachment: Bug 1816992 - Add sentinel attribution to repackage jobs r?bhearsum

Comment 6

[Heitor Neiva [:hneiva]]

Attachment: Bug 1816992 - Add attribution job for DMGs r?bhearsum

Depends on D189404

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/63af389f8fe2
Add sentinel attribution to repackage jobs r=taskgraph-reviewers,releng-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/c06f8d8a7bdc
Add attribution job for DMGs r=taskgraph-reviewers,releng-reviewers,bhearsum

Comment 8

[Natalia Csoregi [:nataliaCs]]

Backed out for causing repackage bustage.

• backout: https://hg.mozilla.org/integration/autoland/rev/a0f40aa5a5952bec0f425366ff9fcec37dd1dceb
• push: https://treeherder.mozilla.org/jobs?repo=autoland&group_state=expanded&revision=c06f8d8a7bdc5622f63413083443f19d02e09f1c&selectedTaskRun=InfGdnlySj-xrSXSYfS4IA.0
• failure log: https://treeherder.mozilla.org/logviewer?job_id=431541478&repo=autoland&lineNumber=696

[task 2023-10-06T04:44:34.390Z] 04:44:34     INFO -  file: /.VolumeIcon.icns
[task 2023-10-06T04:44:34.394Z] 04:44:34     INFO -  Setting permissions to 100644 for /.VolumeIcon.icns
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  Error running mach:
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -      mach --log-no-times repackage dmg --attribution_sentinel __MOZCUSTOM__ --input /builds/worker/fetches/target.tar.gz --output /builds/worker/workspace/outputs/target.dmg
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  The error occurred in code that was called by the mach command. This is either
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  a bug in the called code itself or in the way that mach is calling it.
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  You can invoke ``./mach busted`` to check if this issue is already on file. If it
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  isn't, please use ``./mach busted file repackage`` to report it. If ``./mach busted`` is
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  misbehaving, you can also inspect the dependencies of bug 1543241.
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  If filing a bug, please include the full output of mach, including this error
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  message.
[task 2023-10-06T04:44:34.524Z] 04:44:34     INFO -  The details of the failure are as follows:
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -  subprocess.CalledProcessError: Command '[PosixPath('/builds/worker/fetches/dmg/hfsplus'), '/tmp/tmp55xfa8h7/staged.hfs', 'setattr', 'Firefox Nightly.app', 'com.apple.application-instance', '__MOZCUSTOM__\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t\t']' died with <Signals.SIGSEGV: 11>.
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/mach_commands.py", line 2418, in repackage_dmg
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -      repackage_dmg(input, output, attribution_sentinel)
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozbuild/repackaging/dmg.py", line 47, in repackage_dmg
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -      create_dmg(
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozpack/dmg.py", line 194, in create_dmg
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -      create_dmg_from_staged(
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -    File "/builds/worker/checkouts/gecko/python/mozbuild/mozpack/dmg.py", line 104, in create_dmg_from_staged
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -      subprocess.check_call(
[task 2023-10-06T04:44:34.526Z] 04:44:34     INFO -    File "/usr/lib/python3.11/subprocess.py", line 413, in check_call
[task 2023-10-06T04:44:34.527Z] 04:44:34     INFO -      raise CalledProcessError(retcode, cmd)
[task 2023-10-06T04:44:34.631Z] 04:44:34     INFO - Return code: 1
[task 2023-10-06T04:44:34.631Z] 04:44:34    ERROR - 1 not in success codes: [0]
[task 2023-10-06T04:44:34.631Z] 04:44:34  WARNING - setting return code to 2
[task 2023-10-06T04:44:34.631Z] 04:44:34    FATAL - Halting on failure while running ['/builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/common/bin/python', 'mach', '--log-no-times', 'repackage', 'dmg', '--attribution_sentinel', '__MOZCUSTOM__', '--input', '/builds/worker/fetches/target.tar.gz', '--output', '/builds/worker/workspace/outputs/target.dmg']
[task 2023-10-06T04:44:34.631Z] 04:44:34    FATAL - Running post_fatal callback...
[task 2023-10-06T04:44:34.631Z] 04:44:34    FATAL - Exiting 2
[task 2023-10-06T04:44:34.631Z] 04:44:34     INFO - [mozharness: 2023-10-06 04:44:34.631947Z] Finished repackage step (failed)
[taskcluster 2023-10-06 04:44:35.477Z] === Task Finished ===
[taskcluster 2023-10-06 04:44:35.519Z] Artifact "public/build" not found at "/builds/worker/artifacts/": (HTTP code 404) no such container - Could not find the file /builds/worker/artifacts/ in container 39b965265fd5265f3f84185543df2b827f2f8ca1160612abb48e689a946b323d 
[taskcluster 2023-10-06 04:44:35.520Z] Artifact "public/build/target.dmg" not found at "/builds/worker/workspace/outputs/target.dmg": (HTTP code 404) no such container - Could not find the file /builds/worker/workspace/outputs/target.dmg in container 39b965265fd5265f3f84185543df2b827f2f8ca1160612abb48e689a946b323d 
[taskcluster 2023-10-06 04:44:35.705Z] Unsuccessful task run with exit code: 2 completed in 18.141 seconds

Comment 9

[Heitor Neiva [:hneiva]]

Fixed the problem in autoland, re-landing it now.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7628ecf70973
Add sentinel attribution to repackage jobs r=taskgraph-reviewers,releng-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/1d25ed23e85a
Add attribution job for DMGs r=taskgraph-reviewers,releng-reviewers,bhearsum

Comment 11

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/7628ecf70973
https://hg.mozilla.org/mozilla-central/rev/1d25ed23e85a

Comment 12

[[e-mail] (:bhearsum)]

Attachment: Bug 1816992: continue repackaging mac builds into mar and pkg formats r?#releng-reviewers!

Comment 13

[Atila Butkovits]

Backed out for causing Build bustages.

Backout link: https://hg.mozilla.org/mozilla-central/rev/58f4e6d897080abb0ec8f4cffc3dc8794a863f08

Push where failures started: https://treeherder.mozilla.org/jobs?repo=mozilla-central&resultStatus=testfailed%2Cbusted%2Cexception%2Cretry%2Cusercancel&revision=e9fa7c857657ebfdc8275dc4b60016ed565ebf76&selectedTaskRun=URV_znSmRHu8qyVt4up8Bw.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=431687161&repo=mozilla-central&lineNumber=7

Comment 14

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/591b45ea8b8d
Add sentinel attribution to repackage jobs r=taskgraph-reviewers,releng-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/3e09d9541b69
Add attribution job for DMGs r=taskgraph-reviewers,releng-reviewers,bhearsum
https://hg.mozilla.org/integration/autoland/rev/1069b61535ad
continue repackaging mac builds into mar and pkg formats r=releng-reviewers,gbrown

Comment 15

[Noemi Erli[:noemi_erli]]

https://hg.mozilla.org/mozilla-central/rev/591b45ea8b8d
https://hg.mozilla.org/mozilla-central/rev/3e09d9541b69
https://hg.mozilla.org/mozilla-central/rev/1069b61535ad

Comment 16

[[e-mail] (:bhearsum)]

Attachment: Bug 1816992: pull target.dmg from attribution-l10n jobs r?#releng-reviewers!

This fixes a regression that has caused us to stop publishing new mac installers for localized builds

Comment 17

[Julien Cristau [:jcristau]]

Attachment: Bug 1816992 - restore build_type attribute for attribution-l10n jobs. r?#releng

This was removed presumably by mistake in commit 3e09d954 "Add attribution job
for DMGs"

Comment 18

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/7ededf14ac32
pull target.dmg from attribution-l10n jobs r=releng-reviewers,taskgraph-reviewers,jcristau
https://hg.mozilla.org/integration/autoland/rev/56d67c43bc00
restore build_type attribute for attribution-l10n jobs. r=jlorenzo

Comment 19

[Norisz Fay [:noriszfay]]

https://hg.mozilla.org/mozilla-central/rev/7ededf14ac32
https://hg.mozilla.org/mozilla-central/rev/56d67c43bc00