Closed Bug 2007476 Opened 3 months ago Closed 2 months ago

"OSError: [Errno 86] Bad CPU type in executable: '/Users/*/.mozbuild/android-device/host-utils-145.0a1-en-US.mac/certutil'" on macOS ARM devices

Categories

(Infrastructure & Operations :: RelOps: General, defect)

Product:
Infrastructure & Operations
Component:
RelOps: General
Type:
defect

Tracking

(Not tracked)

Status:
VERIFIED FIXED

People

(Reporter: whimboo, Assigned: kershaw)

References

(Blocks 1 open bug, Regression)

Details

(Keywords: regression)

Attachments

(1 file)

Bug 2007476 - Update mac hostutils.manifest, r=aerickson
48 bytes, text/x-phabricator-request
Details | Review

With the bump of host-utils from version 121 to 145 over on bug 1993177 I can no longer run tests with the Android emulator on my MBP M4 because the architecture of the binaries doesn't match:

CRITICAL Uncaught exception in TestRunnerManager.run:
Traceback (most recent call last):
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 441, in run_loop
    new_state = f()
                ^^^
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/testrunner.py", line 601, in init
    browser = impl.browser_cls(self.logger,
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox_android.py", line 402, in __init__
    super().__init__(logger, config=config, **kwargs)
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 990, in __init__
    self.profile = profile_creator.create()
                   ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 716, in create
    self._setup_ssl(profile)
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 831, in _setup_ssl
    certutil("-N", "-d", cert_db_path, "-f", pw_path)
  File "/Users/henrik/code/gecko/testing/web-platform/tests/tools/wptrunner/wptrunner/browsers/firefox.py", line 818, in certutil
    subprocess.check_output(cmd,
  File "/Users/henrik/.pyenv/versions/3.11.13/lib/python3.11/subprocess.py", line 466, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/.pyenv/versions/3.11.13/lib/python3.11/subprocess.py", line 548, in run
    with Popen(*popenargs, **kwargs) as process:
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/code/gecko/third_party/python/sentry_sdk/sentry_sdk/utils.py", line 1811, in runner
    return sentry_patched_function(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/code/gecko/third_party/python/sentry_sdk/sentry_sdk/integrations/stdlib.py", line 227, in sentry_patched_popen_init
    rv = old_popen_init(self, *a, **kw)
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/henrik/.pyenv/versions/3.11.13/lib/python3.11/subprocess.py", line 1026, in __init__
    self._execute_child(args, executable, preexec_fn, close_fds,
  File "/Users/henrik/.pyenv/versions/3.11.13/lib/python3.11/subprocess.py", line 1955, in _execute_child
    raise child_exception_type(errno_num, err_msg, err_filename)
OSError: [Errno 86] Bad CPU type in executable: '/Users/henrik/.mozbuild/android-device/host-utils-145.0a1-en-US.mac/certutil'

This can be reproduced when eg running web-platform tests via ./mach wpt on an Android emulator.

Andrew, could you please help while Kershaw is out of office? Thanks!

Flags: needinfo?(aerickson)

I wasn't able to reproduce (but I have Rosetta 2 installed). As a temporary workaround, will you please install Rosetta 2 and retry?

softwareupdate --install-rosetta

I ran a bunch of WPT tests on my M4 macbook and never got any Bad CPU type in executable output.

Flags: needinfo?(aerickson)

Thanks. But I'm fine with keeping version 121 of host utils for now (also because it looks like that uninstalling Rosetta is quite hard). That older version seems to still work fine. Also that way I can better verify the fix later on as well.

Flags: needinfo?(kershaw)

Yeah, I can confirm that binaries from host-utils 121 are universal Mach-O (x86_64 + arm64), but the ones from 145 is not.
For example:

../host-utils-121.0a1.en-US.mac/http3server: Mach-O universal binary with 2 architectures: [x86_64:Mach-O 64-bit executable x86_64] [arm64]
../host-utils-121.0a1.en-US.mac/http3server (for architecture x86_64): Mach-O 64-bit executable x86_64
../host-utils-121.0a1.en-US.mac/http3server (for architecture arm64):   Mach-O 64-bit executabl

However, I’m not sure how to produce signed universal binaries. Andrew, maybe you have an idea?

Flags: needinfo?(kershaw) → needinfo?(aerickson)

Because we stopped publishing test artifacts to the FTP server (to save disk space), we aren't able to get a universal build of Firefox on OS X easily.

Bug 1827513 - Stop uploading test packages to archive.mozilla.org

We could combine both architectures (fetched from treeherder) into a single binary with the lipo tool, but it's a fair amount of work.

:whimboo, you're still on an x86 Mac? How long do you think we should maintain support for developers with x86 Macs (with or without Rosetta)?

Flags: needinfo?(aerickson)
Flags: needinfo?(hskupin)

(In reply to Andrew Erickson [:aerickson] from comment #4)

:whimboo, you're still on an x86 Mac? How long do you think we should maintain support for developers with x86 Macs (with or without Rosetta)?

No, I do not have any x86-64 (Intel based) MacBook Pro. As the bug summary says I'm on ARM (M4). Not sure how widespread the Intel-based machines are.

Could we maybe detect the CPU architecture and download whether a ARM or a x86-64 based host-utils package?

Flags: needinfo?(hskupin) → needinfo?(aerickson)
Flags: needinfo?(aerickson)
See Also: → 1827513

Could we maybe detect the CPU architecture and download whether a ARM or a x86-64 based host-utils package?

Yeah, we control mach completely (which installs hostutils)... so that seems doable. We'd have to produce two mac artifacts.

I think the effort to modify mach and make two Mac hostutil builds every time is probably more work than just making a universal mac binary for each hostutils build.

Options that I see:

  • only ship a arm64 binary on OS X
    • abandon x86_64 developers, probably not many... but who knows.
  • only ship a x86_64 binary on OS X
    • forces Mac arm64 developers to install Rosetta 2
  • create a universal binary by hand (with lipo)
    • tied for best solution, but more work
  • ship separate arm64 and x86_64 hostutils packages and modify mach to install correct one
    • tied for best solution, even more work, fragile?

:kershaw, what do you think?

Flags: needinfo?(kershaw)

(In reply to Andrew Erickson [:aerickson] from comment #4)

Because we stopped publishing test artifacts to the FTP server (to save disk space), we aren't able to get a universal build of Firefox on OS X easily.

Bug 1827513 - Stop uploading test packages to archive.mozilla.org

We could combine both architectures (fetched from treeherder) into a single binary with the lipo tool, but it's a fair amount of work.

I don't understand how bug 1827513 is involved here. These artifacts aren't published to archive.mozilla.org anymore, but they're still exposed as tc artifacts from build tasks AFAIK?

(In reply to Andrew Erickson [:aerickson] from comment #7)

Options that I see:

  • only ship a arm64 binary on OS X
    • abandon x86_64 developers, probably not many... but who knows.
  • only ship a x86_64 binary on OS X
    • forces Mac arm64 developers to install Rosetta 2
  • create a universal binary by hand (with lipo)
    • tied for best solution, but more work
  • ship separate arm64 and x86_64 hostutils packages and modify mach to install correct one
    • tied for best solution, even more work, fragile?

:kershaw, what do you think?

I think the ideal solution is to always create universal binaries. As Julien mentioned in comment #9, those artifacts are still exposed on try (for example, this one).

The problem is that the binaries aren’t signed, so even if I download them, they aren’t usable.

Is there a way to generate signed binaries on try?

Flags: needinfo?(kershaw)

(In reply to Julien Cristau [:jcristau] from comment #9)

I don't understand how bug 1827513 is involved here. These artifacts aren't published to archive.mozilla.org anymore, but they're still exposed as tc artifacts from build tasks AFAIK?

archive.mozilla.org had signed universal (arm64 and x86_64) mac binaries. I didn't know treeherder has a universal mac binary target... we asked in matrix firefox-ci and got pointed at an x86_64 build.

Will you please give the treeherder link (vs just the artifact)? What's the heading (e.g. macOS NightlyAsRelease) and target (e.g. B ) ?

Is there a way to generate signed binaries on try?

I don't think so... they need to go through Releng's signing server or be manually signed by Releng (Heitor manually signed the last build).

(In reply to Andrew Erickson [:aerickson] from comment #11)

(In reply to Julien Cristau [:jcristau] from comment #9)

I don't understand how bug 1827513 is involved here. These artifacts aren't published to archive.mozilla.org anymore, but they're still exposed as tc artifacts from build tasks AFAIK?

archive.mozilla.org had signed universal (arm64 and x86_64) mac binaries. I didn't know treeherder has a universal mac binary target... we asked in matrix firefox-ci and got pointed at an x86_64 build.

Will you please give the treeherder link (vs just the artifact)? What's the heading (e.g. macOS NightlyAsRelease) and target (e.g. B ) ?

This is an example treeherder link: https://treeherder.mozilla.org/jobs?repo=mozilla-central&selectedTaskRun=IX0KdISlTom9a5usL7wgqg.0
The heading is macOS Shippable and the target is B in Bpgo.

OK, I've updated https://wiki.mozilla.org/Packaging_Android_host_utilities to point at those builds.

:kershaw, do you want to build a new version or wait until we need an update? I guess we could just do a Mac one.

Flags: needinfo?(kershaw)

It would be great if we could have an update for MacOS. It's very annoying to have to deny the prompt for the update each time I want to run a wpt test on Android. And I cannot see a way to disable the prompt.

(In reply to Andrew Erickson [:aerickson] from comment #13)

OK, I've updated https://wiki.mozilla.org/Packaging_Android_host_utilities to point at those builds.

:kershaw, do you want to build a new version or wait until we need an update? I guess we could just do a Mac one.

Sure. I'll do this.

Flags: needinfo?(kershaw)
Assignee: relops → kershaw
See Also: → 2009659

FYI: I filed bug 2009659 for signing host-utils binaries.

I also have been running into this but my work around is as follows:

  • Create another firefox checkout at git:8055beeb471810c0b4742493854e9a64335ab550 and do a default ./mach build there
  • export MOZ_HOST_BIN=~/projects/firefox-hostutils/obj-aarch64-apple-darwin24.6.0/dist/bin

This git rev is the Fx145 version put in tooltool for intel-macs. I suspect for most things that an up-to-date trunk build is good enough too.

Note that I had to do a full desktop build (no mozconfig file or MOZCONFIG environment variable). The artifacts seem expired for that long ago.

https://hg.mozilla.org/mozilla-central/rev/8ad1a131f5d4

Status: NEW → RESOLVED
Closed: 2 months ago
Resolution: --- → FIXED

This works great now and thankfully I no longer have to accept/deny the upgrade request when running wpt tests on Android.

Thanks a lot for fixing this Kershaw!

Status: RESOLVED → VERIFIED
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: