Open Bug 1781111 Opened 2 months ago Updated 19 days ago

[macOS 13] Mozregression can't run any build on macOS ("Firefox Nightly" is damaged and can't be opened)

Categories

(Testing :: mozregression, defect, P3)

Default
x86_64
macOS
defect

Tracking

(Not tracked)

People

(Reporter: emilio, Assigned: zeid)

References

(Depends on 1 open bug, Blocks 1 open bug)

Details

Attachments

(2 files)

Seems like a code signing error. You get a dialog with no way to override it, only "Move to Bin" and "Cancel".

According to Markus maybe mozregression can remove the quarantine attributes or something? Or maybe we can set them so the download date and so on is ok and maybe you get a "I know what I'm doing button" or so?

Is there a specific time range that this is happening with, or is this happening to all builds? There was an issue reported earlier in the month that seems very similar to this but affected only builds on June 30th -- and likely the issue is with the builds themselves and not mozregression.

Please confirm if this is happening with all/any builds, or with only some builds.

Flags: needinfo?(emilio)

This happens with all builds afaict. I was trying to bisect from 102 which tried to launch builds from May 30th, but trying with --launch 2022-07-25 etc (and various random dates) the same issue happens.

Flags: needinfo?(emilio)

@emilio, I didn't have any problems running mozregression with $ ./mach mozregression --good 102 from an updated mozilla-central repo and I'm wondering how your environment differs. This works for me with Intel or Apple Silicon. $ mozregression --launch 2022-07-25 also works for me.

Could you post some more details to help us reproduce?

  1. OS version
  2. Intel or Apple Silicon (Apple Silicon has different code signing requirements)
  3. How you got mozregression
  4. The full command you're running
Flags: needinfo?(emilio)

(In reply to Haik Aftandilian [:haik] from comment #3)

Could you post some more details to help us reproduce?

  1. OS version

macOS 13 beta

  1. Intel or Apple Silicon (Apple Silicon has different code signing requirements)

Intel

  1. How you got mozregression

./mach mozregression

  1. The full command you're running

./mach mozregression --launch 102.

Flags: needinfo?(emilio)

:emilio -- could you try installing mach via pip (try python3 -m pip install mozregression) and running it directly through the CLI? (i.e. mozregression --debug --launch 102).

If there is any interesting output in the console, please include it as well.

Flags: needinfo?(emilio)
Attached file Mozregression log

Nothing particularly useful there :/

Flags: needinfo?(emilio)

When closing the dialog with "Cancel" I get:

4:43.81 WARNING: Process exited with code -9
Traceback (most recent call last):
  File "/Users/mozilla/Library/Python/3.9/bin/mozregression", line 8, in <module>
    sys.exit(main())
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 346, in main
    sys.exit(method())
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 281, in launch_nightlies
    self._launch(NightlyInfoFetcher)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/main.py", line 278, in _launch
    self.test_runner.run_once(build_info)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/test_runner.py", line 136, in run_once
    return launcher.wait()
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/launchers.py", line 112, in __exit__
    self.cleanup()
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozregression/launchers.py", line 276, in cleanup
    remove(self.tempdir)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 231, in remove
    _update_permissions(os.path.join(root, entry))
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 214, in _update_permissions
    _call_with_windows_retry(os.chmod, (path, mode))
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 191, in _call_with_windows_retry
    _call_windows_retry(*args, **kwargs)
  File "/Users/mozilla/Library/Python/3.9/lib/python/site-packages/mozfile/mozfile.py", line 149, in _call_windows_retry
    func(*args)
PermissionError: [Errno 1] Operation not permitted: '/var/folders/lb/tlnrrts50m13cjtb7yp99s2w0000gn/T/tmpf394cvxm/Firefox Nightly.app'

But nothing too useful there either.

After installing macOS 13 Beta 22A5295i on my Intel Mac, I am able to reproduce the problem.

In the WWDC 2022 macOS "What’s new in privacy" presentation, Apple said they're changing the notarization and signing checks in macOS 13 to be more strict and occur each time the application is run. We are probably being affected by that change, but it doesn't seem to be a problem on Apple Silicon. More debugging is needed to figure out what about our archived builds is causing this. I'll take this bug for now.

From https://developer.apple.com/videos/play/wwdc2022/10096/ :

Gatekeeper checks the integrity of newly-downloaded apps. In macOS Ventura, Gatekeeper will now check the integrity of all notarized apps, not just quarantined apps.

First, apps need to be properly signed. Starting with macOS Ventura, if your notarized app is no longer validly signed, it will be blocked by Gatekeeper on first launch. You should sign all your executables and bundles and ensure that their signatures stay valid when you make changes to your apps. In addition to an integrity check, Gatekeeper will also prevent your app from being modified in certain ways.

Assignee: nobody → haftandilian
Blocks: 1773708

This seems like an issue with the dmg/binaries -- it would be good to confirm that this issue is strictly with the builds and not mozregression (e.g. by downloading https://archive.mozilla.org/pub/firefox/nightly/2022/05/2022-05-30-19-06-26-mozilla-central/firefox-103.0a1.en-US.mac.dmg and extracting/launching manually on the affected platform.)

Type: defect → task
OS: Unspecified → macOS
Priority: -- → P3
Hardware: Unspecified → x86_64

Mozregression, when used with Firefox, downloads autoland builds, if I remember right. And (as far as I know) there are no similar problems with mozilla-central (nightly) builds.

Are these two kinds of builds signed differently? If so, it may be possible to fix this problem by signing autoland builds the same way as mozilla-central builds.

Edit: I just discovered that the autoland builds aren't signed at all.

I just downloaded and ran today's mozilla-central nightly on an Intel VM (VMware Fusion) running macOS 13.0 Beta 3 Update (build 22A5295i). I had no problems.

Anyone know where I can download recent autoland builds? Every attempt to visit http://ftp.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64/ or https://archive.mozilla.org/pub/firefox/tinderbox-builds/autoland-macosx64/ results in a 504 error, after a long delay.

I just used the mozregression gui to download and run the autoland build with build id 20220726112520 (on my macOS 13.0 Intel VM). I had no problems.

https://hg.mozilla.org/integration/autoland/rev/db323adbd984dd62ca8b5c897fa8ebb47663f77d
https://treeherder.mozilla.org/jobs?repo=autoland&revision=1640d9d855e056401800f178476972ec5a864357

However, if I download the 20220726112520 autoland build directly (from the following URL), I get the "damaged" error when I extract it to my desktop and try to run it.

https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UR3qh5mbQbOklwnPLYNI5g/runs/0/artifacts/public/build/target.dmg

There's a workaround for this: Before opening target.dmg, run xattr on it to clear its extended attributes (including any quarantine ones) -- xattr -c target.dmg.

Type: task → defect

If the key to this bug turns out to be the difference between autoland and mozilla-central builds, it probably isn't specific to macOS 13 (Ventura). It should also happen on other (earlier) versions of macOS.

The problem appears to be due to the policies.json file added to the bundle breaking the codesigning check. This is likely to be an issue we have to also resolve for enterprise policies with macOS 13 given that we store them in the .app bundle and they are meant to be modified by an administrator.

See the error at the end of the output below. After removing the policies.json file, and copying the whole bundle downloaded by mozregression to a new location, the codesign check is clean and I am able to launch the browser from the Finder in the usual way. The copying is required because macOS caches notarization results by path.

$ codesign -vvv --deep --strict ./Firefox\ Nightly.app/
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/pingsender
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/pingsender
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/liblgpllibs.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/liblgpllibs.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libipcclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libipcclientcerts.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/firefox-bin
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/firefox-bin
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libfreebl3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libfreebl3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libsoftokn3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libsoftokn3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavutil.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavutil.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/plugin-container.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libosclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libosclientcerts.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/plugin-container.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/crashreporter.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozglue.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozglue.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/crashreporter.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/updater.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnssckbi.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnssckbi.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/updater.app
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavcodec.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libmozavcodec.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/minidump-analyzer
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/minidump-analyzer
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnss3.dylib
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/libnss3.dylib
--prepared:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/XUL
--validated:/Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/MacOS/XUL
./Firefox Nightly.app/: a sealed resource is missing or invalid
file added: /Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/Resources/distribution/policies.json
Summary: Mozregression can't run any build on macOS ("Firefox Nightly" is damaged and can't be opened) → [macOS 13] Mozregression can't run any build on macOS ("Firefox Nightly" is damaged and can't be opened)

This issue that :haik mentioned above

file added: /Users/haftandilian/Desktop/tmpzyhlh1xb/Firefox Nightly.app/Contents/Resources/distribution/policies.json

is due to the launcher adding the policies.json file in the <app>/Contents/Resources/distribution directory upon launch, for the purpose of disabling automatic app updates (see this code). Whatever solution is implemented in Firefox will have to be reflected in mozregression as well to ensure that the DisableAppUpdate can be set.

See Also: → 1777554

I installed the Python3 mozregression package (using python3 -m pip install mozregression) and then ran it as follows:

~/Library/Python/3.9/bin/mozregression --launch 102

No problems.

Presumably, in this case, the launcher didn't add a policies.json file. Can someone please tell me how to trigger this behavior? I have a nagging doubt that this problem isn't specific to macOS 13, and I'd like to be able to test my hunch.

Presumably, in this case, the launcher didn't add a policies.json file.

No, it did, and codesign -vvv --deep --strict showed the "sealed resource is missing or invalid" error on it. And I still had no problems.

(In reply to Steven Michaud [:smichaud] (Retired) from comment #18)

Presumably, in this case, the launcher didn't add a policies.json file. Can someone please tell me how to trigger this behavior? I have a nagging doubt that this problem isn't specific to macOS 13, and I'd like to be able to test my hunch.

:smichaud -- is it possible that this is not impacting everyone in the same way depending on system settings? :haik reproduced the issue reliably and we tested commenting out the line that writes policies.json in mozregression, which suppressed this issue reliably as well.

No, it did, and codesign -vvv --deep --strict showed the "sealed resource is missing or invalid" error on it. And I still had no problems.

Another thing you could check is, within a Python shell, you should see this output:

>>> import mozinfo
>>> mozinfo.os
'mac'

And this is what determines the location of policies.json, if for whatever reason you're getting a different output, it's possible the file ended up in a different location that does not trigger this error?

Lastly, it's possible that there is more than one version of macOS 13 beta, so perhaps that has something to do with it.

I figured out why my results are different with ~/Library/Python/3.9/bin/mozregression --launch 102 -- I had SIP disabled (using csrutil disable in the recovery partition). I get the same results as Haik (and everyone else) after I re-enable it.

Sorry for the confusion :-(

For what it's worth, I've now tried mozregression --launch 102 on both macOS 12.5 and macOS 11.6.8, with SIP enabled. I didn't have any problems, and in both cases the policies.json file was added and triggered an error from codesign -vvv --deep --strict.

So yes, this bug is specific to macOS 13.

Depends on: 1781462

@emilio, could you confirm the error message from comment 8 no longer occurs if you going into macOS System Preferences -> Privacy & Security -> App Management and add Terminal to the list and allow it. Firefox will still fail to launch, but that permission error should go away. That error looks like a distinct problem that might need a mozregression fix.

Flags: needinfo?(emilio)

For what it's worth mozregression --launch 102 also fails (with the "damaged" error) on an Apple Silicon machine running macOS 13.0 Beta 3 (build 22A5295i) with SIP enabled (csrutil enable).

Assignee: haftandilian → zeid
See Also: → 1781813

(In reply to Haik Aftandilian [:haik] from comment #23)

@emilio, could you confirm the error message from comment 8 no longer occurs if you going into macOS System Preferences -> Privacy & Security -> App Management and add Terminal to the list and allow it. Firefox will still fail to launch, but that permission error should go away. That error looks like a distinct problem that might need a mozregression fix.

Yeah (well, I added iTerm rather than Terminal because that's what I use), but that error message goes away indeed.

Flags: needinfo?(emilio)

The severity field is not set for this bug.
:zeid, could you have a look please?

For more information, please visit auto_nag documentation.

Flags: needinfo?(zeid)
Severity: -- → S2
Flags: needinfo?(zeid)
No longer blocks: 1779800
See Also: → 1779800
You need to log in before you can comment on or make changes to this bug.