1781111 - [macOS 13 Ventura] Mozregression can't run some builds on macOS ("Firefox Nightly" is damaged and can't be opened)

Status: RESOLVED FIXED

(Testing :: mozregression, defect, P2)

Description

[Emilio Cobos Álvarez (:emilio)]

Seems like a code signing error. You get a dialog with no way to override it, only "Move to Bin" and "Cancel".

According to Markus maybe mozregression can remove the quarantine attributes or something? Or maybe we can set them so the download date and so on is ok and maybe you get a "I know what I'm doing button" or so?

Comment 1

[Zeid Zabaneh [:zeid]]

Is there a specific time range that this is happening with, or is this happening to all builds? There was an issue reported earlier in the month that seems very similar to this but affected only builds on June 30th -- and likely the issue is with the builds themselves and not mozregression.

Please confirm if this is happening with all/any builds, or with only some builds.

Comment 2

[Emilio Cobos Álvarez (:emilio)]

This happens with all builds afaict. I was trying to bisect from 102 which tried to launch builds from May 30th, but trying with --launch 2022-07-25 etc (and various random dates) the same issue happens.

Comment 3

[Haik Aftandilian [:haik]]

@emilio, I didn't have any problems running mozregression with $ ./mach mozregression --good 102 from an updated mozilla-central repo and I'm wondering how your environment differs. This works for me with Intel or Apple Silicon. $ mozregression --launch 2022-07-25 also works for me.

Could you post some more details to help us reproduce?

1. OS version
2. Intel or Apple Silicon (Apple Silicon has different code signing requirements)
3. How you got mozregression
4. The full command you're running

Comment 4

[Emilio Cobos Álvarez (:emilio)]

(In reply to Haik Aftandilian [:haik] from comment #3)

Could you post some more details to help us reproduce?

1. OS version

macOS 13 beta

2. Intel or Apple Silicon (Apple Silicon has different code signing requirements)

Intel

2. How you got mozregression

./mach mozregression

3. The full command you're running

./mach mozregression --launch 102.

Comment 5

[Zeid Zabaneh [:zeid]]

:emilio -- could you try installing mach via pip (try python3 -m pip install mozregression) and running it directly through the CLI? (i.e. mozregression --debug --launch 102).

If there is any interesting output in the console, please include it as well.

Comment 6

[Emilio Cobos Álvarez (:emilio)]

Attachment: Mozregression log

Nothing particularly useful there :/

Comment 7

[Emilio Cobos Álvarez (:emilio)]

Attachment: Screenshot of the dialog

Comment 8

[Emilio Cobos Álvarez (:emilio)]

When closing the dialog with "Cancel" I get:

4:43.81 WARNING: Process exited with code -9
Traceback (most recent call last):
  File "/Users/mozilla/Library/Python/3.9/bin/mozregression", line 8, in <module>
    sys.exit(main())
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 346, in main
    sys.exit(method())
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 281, in launch_nightlies
    self._launch(NightlyInfoFetcher)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 278, in _launch
    self.test_runner.run_once(build_info)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/test_runner.py", line 136, in run_once
    return launcher.wait()
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/launchers.py", line 112, in __exit__
    self.cleanup()
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/launchers.py", line 276, in cleanup
    remove(self.tempdir)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 231, in remove
    _update_permissions(os.path.join(root, entry))
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 214, in _update_permissions
    _call_with_windows_retry(os.chmod, (path, mode))
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 191, in _call_with_windows_retry
    _call_windows_retry(*args, **kwargs)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 149, in _call_windows_retry
    func(*args)
PermissionError: [Errno 1] Operation not permitted: '/var/folders/lb/tlnrrts50m13cjtb7yp99s2w0000gn/T/tmpf394cvxm/Firefox Nightly.app'

But nothing too useful there either.

Comment 9

[Haik Aftandilian [:haik]]

After installing macOS 13 Beta 22A5295i on my Intel Mac, I am able to reproduce the problem.

In the WWDC 2022 macOS "What’s new in privacy" presentation, Apple said they're changing the notarization and signing checks in macOS 13 to be more strict and occur each time the application is run. We are probably being affected by that change, but it doesn't seem to be a problem on Apple Silicon. More debugging is needed to figure out what about our archived builds is causing this. I'll take this bug for now.

From https://developer.apple.com/videos/play/wwdc2022/10096/ :

Gatekeeper checks the integrity of newly-downloaded apps. In macOS Ventura, Gatekeeper will now check the integrity of all notarized apps, not just quarantined apps.

First, apps need to be properly signed. Starting with macOS Ventura, if your notarized app is no longer validly signed, it will be blocked by Gatekeeper on first launch. You should sign all your executables and bundles and ensure that their signatures stay valid when you make changes to your apps. In addition to an integrity check, Gatekeeper will also prevent your app from being modified in certain ways.

Comment 10

[Zeid Zabaneh [:zeid]]

This seems like an issue with the dmg/binaries -- it would be good to confirm that this issue is strictly with the builds and not mozregression (e.g. by downloading https://archive.mozilla.org/pub/firefox/nightly/2022/05/2022-05-30-19-06-26-mozilla-central/firefox-103.0a1.en-US.mac.dmg and extracting/launching manually on the affected platform.)

Comment 11

[Steven Michaud [:smichaud] (Retired)]

Mozregression, when used with Firefox, downloads autoland builds, if I remember right. And (as far as I know) there are no similar problems with mozilla-central (nightly) builds.

Are these two kinds of builds signed differently? If so, it may be possible to fix this problem by signing autoland builds the same way as mozilla-central builds.

Edit: I just discovered that the autoland builds aren't signed at all.

Comment 12

[Steven Michaud [:smichaud] (Retired)]

I just downloaded and ran today's mozilla-central nightly on an Intel VM (VMware Fusion) running macOS 13.0 Beta 3 Update (build 22A5295i). I had no problems.

Anyone know where I can download recent autoland builds? Every attempt to visit http://ftp.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64/ or https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64/ results in a 504 error, after a long delay.

Comment 13

[Steven Michaud [:smichaud] (Retired)]

I just used the mozregression gui to download and run the autoland build with build id 20220726112520 (on my macOS 13.0 Intel VM). I had no problems.

https://hg.mozilla.org/integration/autoland/rev/db323adbd984dd62ca8b5c897fa8ebb47663f77d
https://treeherder.mozilla.org/jobs?repo=autoland&revision=1640d9d855e056401800f178476972ec5a864357

Comment 14

[Steven Michaud [:smichaud] (Retired)]

However, if I download the 20220726112520 autoland build directly (from the following URL), I get the "damaged" error when I extract it to my desktop and try to run it.

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UR3qh5mbQbOklwnPLYNI5g/runs/0/artifacts/public/build/target.dmg

There's a workaround for this: Before opening target.dmg, run xattr on it to clear its extended attributes (including any quarantine ones) -- xattr -c target.dmg.

Comment 15

[Steven Michaud [:smichaud] (Retired)]

If the key to this bug turns out to be the difference between autoland and mozilla-central builds, it probably isn't specific to macOS 13 (Ventura). It should also happen on other (earlier) versions of macOS.

Comment 16

[Haik Aftandilian [:haik]]

The problem appears to be due to the policies.json file added to the bundle breaking the codesigning check. This is likely to be an issue we have to also resolve for enterprise policies with macOS 13 given that we store them in the .app bundle and they are meant to be modified by an administrator.

See the error at the end of the output below. After removing the policies.json file, and copying the whole bundle downloaded by mozregression to a new location, the codesign check is clean and I am able to launch the browser from the Finder in the usual way. The copying is required because macOS caches notarization results by path.

$ codesign -vvv --deep --strict ./Firefox\ Nightly.app/
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/pingsender
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/pingsender
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/liblgpllibs.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/liblgpllibs.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libipcclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libipcclientcerts.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/firefox-bin
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/firefox-bin
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libfreebl3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libfreebl3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libsoftokn3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libsoftokn3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavutil.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavutil.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/plugin-container.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libosclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libosclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/plugin-container.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/crashreporter.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozglue.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozglue.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/crashreporter.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/updater.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnssckbi.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnssckbi.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/updater.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavcodec.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavcodec.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/minidump-analyzer
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/minidump-analyzer
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnss3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnss3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/XUL
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/XUL
./Firefox Nightly.app/: a sealed resource is missing or invalid
file added: /Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/Resources/distribution/policies.json

Comment 17

[Zeid Zabaneh [:zeid]]

This issue that :haik mentioned above

file added: /Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/Resources/distribution/policies.json

is due to the launcher adding the policies.json file in the <app>/Contents/Resources/distribution directory upon launch, for the purpose of disabling automatic app updates (see this code). Whatever solution is implemented in Firefox will have to be reflected in mozregression as well to ensure that the DisableAppUpdate can be set.

Comment 18

[Steven Michaud [:smichaud] (Retired)]

I installed the Python3 mozregression package (using python3 -m pip install mozregression) and then ran it as follows:

~/Library/Python/3.9/bin/mozregression --launch 102

No problems.

Presumably, in this case, the launcher didn't add a policies.json file. Can someone please tell me how to trigger this behavior? I have a nagging doubt that this problem isn't specific to macOS 13, and I'd like to be able to test my hunch.

Comment 19

[Steven Michaud [:smichaud] (Retired)]

Presumably, in this case, the launcher didn't add a policies.json file.

No, it did, and codesign -vvv --deep --strict showed the "sealed resource is missing or invalid" error on it. And I still had no problems.

Comment 20

[Zeid Zabaneh [:zeid]]

(In reply to Steven Michaud [:smichaud] (Retired) from comment #18)

Presumably, in this case, the launcher didn't add a policies.json file. Can someone please tell me how to trigger this behavior? I have a nagging doubt that this problem isn't specific to macOS 13, and I'd like to be able to test my hunch.

:smichaud -- is it possible that this is not impacting everyone in the same way depending on system settings? :haik reproduced the issue reliably and we tested commenting out the line that writes policies.json in mozregression, which suppressed this issue reliably as well.

No, it did, and codesign -vvv --deep --strict showed the "sealed resource is missing or invalid" error on it. And I still had no problems.

Another thing you could check is, within a Python shell, you should see this output:

>>> import mozinfo
>>> mozinfo.os
'mac'

And this is what determines the location of policies.json, if for whatever reason you're getting a different output, it's possible the file ended up in a different location that does not trigger this error?

Lastly, it's possible that there is more than one version of macOS 13 beta, so perhaps that has something to do with it.

Comment 21

[Steven Michaud [:smichaud] (Retired)]

I figured out why my results are different with ~/Library/Python/3.9/bin/mozregression --launch 102 -- I had SIP disabled (using csrutil disable in the recovery partition). I get the same results as Haik (and everyone else) after I re-enable it.

Sorry for the confusion :-(

Comment 22

[Steven Michaud [:smichaud] (Retired)]

For what it's worth, I've now tried mozregression --launch 102 on both macOS 12.5 and macOS 11.6.8, with SIP enabled. I didn't have any problems, and in both cases the policies.json file was added and triggered an error from codesign -vvv --deep --strict.

So yes, this bug is specific to macOS 13.

Comment 23

[Haik Aftandilian [:haik]]

@emilio, could you confirm the error message from comment 8 no longer occurs if you going into macOS System Preferences -> Privacy & Security -> App Management and add Terminal to the list and allow it. Firefox will still fail to launch, but that permission error should go away. That error looks like a distinct problem that might need a mozregression fix.

Comment 24

[Steven Michaud [:smichaud] (Retired)]

For what it's worth mozregression --launch 102 also fails (with the "damaged" error) on an Apple Silicon machine running macOS 13.0 Beta 3 (build 22A5295i) with SIP enabled (csrutil enable).

Comment 25

[Emilio Cobos Álvarez (:emilio)]

(In reply to Haik Aftandilian [:haik] from comment #23)

@emilio, could you confirm the error message from comment 8 no longer occurs if you going into macOS System Preferences -> Privacy & Security -> App Management and add Terminal to the list and allow it. Firefox will still fail to launch, but that permission error should go away. That error looks like a distinct problem that might need a mozregression fix.

Yeah (well, I added iTerm rather than Terminal because that's what I use), but that error message goes away indeed.

Comment 26

[BugBot [:suhaib / :marco/ :calixte]]

The severity field is not set for this bug.
:zeid, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 27

[Zeid Zabaneh [:zeid]]

Changing the title as it seems that bug 1767139 reduced the number of builds that are failing to launch on Ventura, although this is still reproducible on other builds.

Locally, these are two examples:

Successful launch:

mozregression --launch ac1330b68d3e7b231a177cfa1ac52e1b2199bb84 --debug
...
 0:01.72 INFO: Running mozilla-central build built on 2022-10-18 05:50:46.941000, revision ac1330b6
 0:10.73 DEBUG: codesign verify exit code: 1
 0:10.73 DEBUG: codesign verification failed for /private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpx0ndpf6e/Firefox Nightly.app, resigning...
/private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpx0ndpf6e/Firefox Nightly.app: replacing existing signature
 0:11.69 INFO: Launching /private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpx0ndpf6e/Firefox Nightly.app/Contents/MacOS/firefox
 0:11.69 INFO: Application command: /private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpx0ndpf6e/Firefox Nightly.app/Contents/MacOS/firefox -foreground -profile /var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmp362nih_e.mozrunner
 0:11.70 INFO: application_buildid: 20221017213658
 0:11.70 INFO: application_changeset: ac1330b68d3e7b231a177cfa1ac52e1b2199bb84
 0:11.70 INFO: application_name: Firefox
 0:11.70 INFO: application_repository: https://hg.mozilla.org/mozilla-central
 0:11.70 INFO: application_version: 108.0a1
...

Failed launch:

mozregression --launch 107 --debug
...
 0:00.76 INFO: Running mozilla-central build for 2022-10-17
 0:09.91 DEBUG: codesign verify exit code: 0
 0:09.91 INFO: Launching /private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpfbj6gm9g/Firefox Nightly.app/Contents/MacOS/firefox
 0:09.91 INFO: Application command: /private/var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpfbj6gm9g/Firefox Nightly.app/Contents/MacOS/firefox -foreground -profile /var/folders/mz/dt2wvcqn2v18chcgg04003k40000gn/T/tmpuezxv1v8.mozrunner
 0:09.93 INFO: application_buildid: 20221017213658
 0:09.93 INFO: application_changeset: ac1330b68d3e7b231a177cfa1ac52e1b2199bb84
 0:09.93 INFO: application_name: Firefox
 0:09.93 INFO: application_repository: https://hg.mozilla.org/mozilla-central
 0:09.93 INFO: application_version: 108.0a1
...

I'm on arm64 (M1) running macOS 13.0.

Comment 28

[Zeid Zabaneh [:zeid]]

Attachment: GitHub Pull Request

Comment 29

[Zeid Zabaneh [:zeid]]

Note that the workaround in pull request 1142 will obfuscate any potential bugs/bug fixes that are related to code signing. Although on the surface this does not seem to be applicable to bug 1799332 since for this bug, the problem is related to a code sign issue after an update (and updates are disabled in mozregression for now).

Somewhat related to bug 1781813 -- we should look into possible alternative ways of disabling updates for future versions of Firefox that are do not interfere with code signatures and can handle multiple installations (e.g., an environment variable).

Comment 30

[Sören Hentzschel]

Still doesn't work for me with mozregression 5.2.0 on macOS 13.0.1, tested on two different MacBooks.

Comment 31

[Zeid Zabaneh [:zeid]]

(In reply to Sören Hentzschel from comment #30)

Still doesn't work for me with mozregression 5.2.0 on macOS 13.0.1, tested on two different MacBooks.

Sören -- can you please attach a log (if you can also run with --debug) as well as what the exact error/symptom is?

Comment 32

[:glob ✱]

Re-marking as fixed as the code has landed and been deployed. If we need to fix things in follow-up lets do that in a new bug please.

Comment 33

[Sören Hentzschel]

(In reply to Zeid Zabaneh [:zeid] from comment #31)

Sören -- can you please attach a log (if you can also run with --debug) as well as what the exact error/symptom is?

I justed managed to install the command line tool and it works! I previously used the mozregression GUI and this is still broken. I filed bug 1802359 for that.