Open Bug 1588081 Opened 6 months ago Updated 4 months ago

Signed geckodriver binaries cannot be started on MacOS 10.15 (Catalina)

Categories

(Testing :: geckodriver, defect, P1, blocker)

Version 3
defect

Tracking

(Not tracked)

People

(Reporter: whimboo, Unassigned)

References

(Blocks 2 open bugs, )

Details

With bug 1546299 we got signed builds of geckodriver, and given its dependency on bug 1471004 the notarization service should have been used. But the geckodriver binary cannot be started on MacOS 10.15 as of today:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=geckodriver&selectedJob=270854573

Aki and Johann, can you please have a look what's broken here? Right now it means users on Catalina cannot run automated tests via Selenium. So I hope that we can find a solution hopefully soon. Thanks!

Flags: needinfo?(jlorenzo)
Flags: needinfo?(aki)

Hey Henrik! Thanks for looping us in. Do you have an error or a screenshot? I haven't migrated my machine to Catalina yet, so I can't reproduce.

Flags: needinfo?(jlorenzo)

Andreas migrated to Catalina, so he has.

Also have a look at https://github.com/mozilla/geckodriver/issues/1629 where I got the information initially.

Oh no, disaster! I haven't upgraded to 10.15 and won't for a jolly long while so I can't help investigate here. Hopefully things are clear. Haik is very knowledgable about these things, I think, and may be able to assist in a pinch.

Ah, we never added notarization for the iscript sign_geckodriver behavior. We can add it, but we will only be able to notarize geckodriver with the nightly- or release-signing certs. Likely that will mean we should stick with geckodriver signing in nightly graphs.

Flags: needinfo?(aki)

Aki, would you mind explaining why we have this hard requirement on nightly or release? We thought that we can build (already done) and sign (bug 1577110) geckodriver based on toolchain builds, which are way faster and better to handle. Now it looks like that due to notarization we were working towards a dead-end.

Flags: needinfo?(ato) → needinfo?(aki)
Priority: -- → P1

Apple will only notarize a build that has been signed with a valid [shipping] key, like the nightly or release keys. Also, it's a security hole to just allow people to sign any arbitrary binary with the nightly or release keys, so we have checks in place to make sure the requests come from a valid nightly or release graph.

Flags: needinfo?(aki)
Blocks: 1573798
Blocks: 1577110

So I think what I miss is the difference to the chain-of-trust requirement. Given that toolchain jobs ensure to keep that, and as given by Johan we should be able to sign also non-nightly and non-release builds on mozilla-central. But what makes notarization different to that?

Flags: needinfo?(aki)

Ahh, sorry. You're right; I'm wrong. If the task runs on an on-push or action graph on mozilla-central with CoT enabled, we should be able to nightly-sign and notarize geckodriver. Apologies for the confusion!

Flags: needinfo?(aki)

Puh! This is lovely to hear! Is there anything else I can help with? What do you think how long it might take to get the notarization working for the geckodriver signing? We just released the 0.26.0 release, but might be able to just replace the binary with the new one on the github releases page.

Flags: needinfo?(aki)

Hm, getting

Downloaded ticket has been stored at file:///var/folders/z4/f2_cz9d95jdctsrtnr3016zm00001_/T/6c927b89-3903-421a-815e-3952f32b6a89.ticket.
Could not remove existing ticket from geckodriver/Contents/CodeResources -- file:///builds/scriptworker/aki/work/0/ because an error occurred. Error Domain=NSCocoaErrorDomain Code=512 "“CodeResources” couldn’t be removed." UserInfo={NSFilePath=/builds/scriptworker/aki/work/0/geckodriver/Contents/CodeResources, NSUserStringVariant=(
    Remove
), NSUnderlyingError=0x7f9a85d85c10 {Error Domain=NSPOSIXErrorDomain Code=20 "Not a directory"}}
The staple and validate action failed! Error 73.

geckodriver is a file, not a directory. I wonder if we have to create a dmg or pkg first, and notarize that?

Aki, might the info from https://developer.apple.com/documentation/security/notarizing_your_app_before_distribution/resolving_common_notarization_issues help here?

In this document they talk about binary or app bundle. So I assume notarizing binaries should be possible, and maybe the code which is currently in use by us to do that, doesn't take just binaries into account.

That document points out:

If you see error -73 while using Xcode 10, you can resolve this issue by upgrading to Xcode 10.1 or later. Also, in this case, check to ensure that the disk image or flat installer package you’re notarizing is writable so you can attach the ticket to the package with the stapler utility.

(emphasis mine).

We're already using Xcode 10.2.x. However, we're not notarizing a disk image or flat installer package; we're trying to notarize a single executable binary. We likely either need to notarize properly, which probably means creating a dmg package containing the executable, or we need to bypass the security restrictions on machines that run geckodriver.

What about a zip file as we offer it for download? While stapling to a binary is currently not possible (see the below link), notarizing the zip archive might be somewhat doable? But I do not really fully understand the paragraph given that all items have to be stapled:

https://developer.apple.com/documentation/xcode/notarizing_your_app_before_distribution/customizing_the_notarization_workflow#3087720

I assume that both the unsigned and signed versions of geckodriver are affected by that?

Aiui, the link says you need to staple everything inside the zipfile. We can't staple a bare geckodriver, so putting it in a zipfile doesn't help our stapling situation. Is there someone who can create a geckodriver.app for mac? We can notarize an app.

As a workaround, users may be able to run xattr to remove the quarantine attribute from geckodriver:
https://derflounder.wordpress.com/2012/11/20/clearing-the-quarantine-extended-attribute-from-downloaded-applications/

Good information. Andreas, would you mind to test if that works for you on 10.15?

Flags: needinfo?(ato)

On macOS 10.15 (Catalina) with geckodriver 0.26.0:

% xattr geckodriver
com.apple.macl
com.apple.quarantine
% ./geckodriver
Killed: 9
% xattr -r -d com.apple.quarantine geckodriver
% xattr geckodriver
com.apple.macl
% ./geckodriver
^C

So it seems users can use xattr(1) to remove the quarantine check manually.

Flags: needinfo?(ato)

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #15)

Aiui, the link says you need to staple everything inside the
zipfile. We can't staple a bare geckodriver, so putting it in a
zipfile doesn't help our stapling situation. Is there someone who
can create a geckodriver.app for mac? We can notarize an app.

Thank you, Aki, for investigating this!

What I have trouble understanding is how Apple is expecting
non-interactive, “terminal-based” programs to be distributed under
the new notarisation regime. geckodriver is not a program it is
suitable to distribute as an application bundle (.app) because it
is not an interactive program like Firefox.

I have a few perhaps stupid questions:

  1. If we could distribute a .dmg instead of a .zip, would we be
    able to notarise the .dmg?

  2. If we can’t notarise .dmg’s, it is more suitable for us to
    distribute geckodriver in a .pkg installer. Do you have any experience
    crafting one? Presumably we can do this in automation too.

(In reply to Andreas Tolfsen 「:ato」 from comment #19)

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #15)

Aiui, the link says you need to staple everything inside the
zipfile. We can't staple a bare geckodriver, so putting it in a
zipfile doesn't help our stapling situation. Is there someone who
can create a geckodriver.app for mac? We can notarize an app.

Thank you, Aki, for investigating this!

What I have trouble understanding is how Apple is expecting
non-interactive, “terminal-based” programs to be distributed under
the new notarisation regime. geckodriver is not a program it is
suitable to distribute as an application bundle (.app) because it
is not an interactive program like Firefox.

I'm not sure. But we're able to launch executables under an .app via commandline, e.g. mkdir /tmp/foo && /Applications/Firefox.app/Contents/MacOS/firefox --profile /tmp/foo

I have a few perhaps stupid questions:

  1. If we could distribute a .dmg instead of a .zip, would we be
    able to notarise the .dmg?

We notarize Firefox.app that's in a .dmg currently, but it's an .app , and we extract it from the dmg before signing and notarizing. I think a bare geckodriver in a .dmg would hit the same problems.

  1. If we can’t notarise .dmg’s, it is more suitable for us to
    distribute geckodriver in a .pkg installer. Do you have any experience
    crafting one? Presumably we can do this in automation too.

We do create a .pkg using pkgbuild, but it's from the existing .app structure. We may be able to create a .pkg with just geckodriver and the appropriate directory structure, but I'm not 100% sure if that will notarize by itself or not. If we're able to create that pkg, we'd need to install it, and then run geckodriver from the installed location.

It looks like our options are:

  • use xattr and run geckodriver from commandline
  • create an .app and notarize it; install the .app and run geckodriver from commandline
  • create a .pkg and notarize it; install the .pkg and run geckodriver from commandline
    • this may or may not require creating an .app first
Flags: needinfo?(aki)

I found this in the Apple documentation:

Alternatively, you can put apps, kernel extensions, and other
software in a container, like a disk image, and notarize the
container. The notary service accepts disk images (UDIF format),
signed flat installer packages, and ZIP archives. It processes
nested containers as well, like packages inside a disk image.

For geckodriver on macOS we currently produce a gunzipped tarball
(.tar.gz), but according to this it should be possible to notarise
.dmg and .zip archives.

As I’m completely unaware of where any of the packaging code lives,
could someone point me to the right resources to try out this change?

No longer blocks: 1573798
Severity: normal → blocker

Please also note that the notarization issue for users only occurs when the binary gets downloaded interactively by the user via a web browser. Then the quarantined bit gets set. If the download happens with eg. curl or wget instead, the binary can directly be used.

Aha, that sounds familiar. Since we have both the curl/wget option, and the xattr option, are we good here?

I documented the problem and the workarounds in https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html, then updated the changelog.

If we can somewhat easily change the geckodriver packaging to .dmg and .dmgs indeed can be notarised, I would suggest we make that change before the 0.27.0 release. I haven’t followed up on your suggestions from comment #22 as I’m struggling to figure out where the packaging is actually made. I should add I’m not familiar with TaskCluster at all.

(In reply to Andreas Tolfsen 「:ato」 from comment #25)

I documented the problem and the workarounds in https://firefox-source-docs.mozilla.org/testing/geckodriver/Notarization.html, then updated the changelog.

If we can somewhat easily change the geckodriver packaging to .dmg and .dmgs indeed can be notarised, I would suggest we make that change before the 0.27.0 release. I haven’t followed up on your suggestions from comment #22 as I’m struggling to figure out where the packaging is actually made. I should add I’m not familiar with TaskCluster at all.

The packaging is all done here: https://searchfox.org/mozilla-central/rev/55aa17110091deef24b913d033ccaf58f9c6d337/taskcluster/scripts/misc/build-geckodriver.sh#56-59.

(In reply to Aki Sasaki [:aki] (he/him) (UTC-7) from comment #22)

We have, e.g., https://searchfox.org/mozilla-central/source/build/package/mac_osx/make-diskimage , https://searchfox.org/mozilla-central/source/python/mozbuild/mozpack/dmg.py . Found via https://searchfox.org/mozilla-central/search?q=DMG_&path= .

make-diskimage appears unused. Looking at dmg.py, I expect the trickiest part will be arranging for all of the macOS-specific tools to be available on the Linux builders (everything is cross compiled in the toolchain tasks). It might be that the existing builder toolchains for cross compiling Just Work, though :)

You need to log in before you can comment on or make changes to this bug.