Closed Bug 1744197 Opened 3 years ago Closed 3 years ago

ERROR: Cannot find a wasi sysroot. Please give its location with --with-wasi-sysroot. on mozilla-release

Categories

(Firefox Build System :: Bootstrap Configuration, defect)

Product:
Firefox Build System
Component:
Bootstrap Configuration
Type:
defect

Tracking

(Not tracked)

Status:
RESOLVED WONTFIX

People

(Reporter: arai, Assigned: ahochheiden)

References

Details

confirmed on linux

Steps to reproduce:

  1. remove ~/.mozbuild
  2. clone mozilla-unified
  3. checkout d7ba6bbdc41bf0c76d05937c5104894d98637d92 revision in release branch
  4. ./mach bootstrap with option 2
  5. ./mach clobber
  6. ./mach build with no mozconfig

Actual result:
Build fails with

 0:03.13 checking for ice sm... yes
 0:03.14 checking MOZ_X11_SM_CFLAGS...
 0:03.14 checking for nasm... /usr/bin/nasm
 0:03.14 checking nasm version... 2.15.05
 0:03.15 ERROR: Cannot find a wasi sysroot. Please give its location with --with-wasi-sysroot. Or build with --without-wasm-sandboxed-libraries.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/home/arai/.mozbuild/_virtualenvs/mach/bin/python', '/home/arai/projects/mozilla-unified/configure.py']

  File "/home/arai/projects/mozilla-unified/python/mozbuild/mozbuild/build_commands.py", line 220, in build
    return driver.build(
  File "/home/arai/projects/mozilla-unified/python/mozbuild/mozbuild/controller/building.py", line 1145, in build
    config_rc = self.configure(
  File "/home/arai/projects/mozilla-unified/python/mozbuild/mozbuild/controller/building.py", line 1526, in configure
    status = self._run_command_in_objdir(
  File "/home/arai/projects/mozilla-unified/python/mozbuild/mozbuild/base.py", line 853, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/home/arai/projects/mozilla-unified/python/mach/mach/mixin/process.py", line 174, in run_process
    raise Exception(

Expected result:
no error

Other notes:
tried some other revisions, and looks like there are 3 patterns:

  • (a) installs sysroot-wasm32-wasi and works (e.g. d8d4af1e17aeb5bc0c4a801cb37e13d9a0171deb)
  • (b) doesn't install sysroot-wasm32-wasi and fails (e.g., d7ba6bbdc41bf0c76d05937c5104894d98637d92)
  • (c) doesn't install sysroot-wasm32-wasi and works (e.g., 781c516b3998c3b7c014e762a970ffd113c2a5b5)
Component: Toolchains → Bootstrap Configuration

on pattern (a), I see the following output in ./mach build

 0:21.52 checking nasm version... 2.15.05
 0:21.52 Installing bootstrapped toolchain in /home/arai/.mozbuild/sysroot-wasm32-wasi
 0:23.21 Setting up artifact sysroot-wasm32-wasi.tar.zst
 0:23.21 Downloading artifact to local cache: /home/arai/.mozbuild/toolchains/8dc5e797af4ea74e-sysroot-wasm32-wasi.tar.zst
 0:23.33 Downloading... 0.0 %
 0:23.35 Downloading... 5.3 %
 0:23.36 Downloading... 10.6 %
 0:23.39 Downloading... 15.1 %
 0:23.39 Downloading... 20.4 %
 0:23.40 Downloading... 25.7 %
 0:23.42 Downloading... 30.1 %
 0:23.44 Downloading... 35.4 %
 0:23.44 Downloading... 40.8 %
 0:23.48 Downloading... 45.2 %
 0:23.48 Downloading... 50.5 %
 0:23.48 Downloading... 55.8 %
 0:23.49 Downloading... 60.3 %
 0:23.50 Downloading... 65.6 %
 0:23.50 Downloading... 70.0 %
 0:23.51 Downloading... 75.3 %
 0:23.52 Downloading... 80.6 %
 0:23.53 Downloading... 85.1 %
 0:23.54 Downloading... 90.4 %
 0:23.55 Downloading... 95.7 %
 0:23.56 Downloading... 100.0 %
 0:23.57 untarring "/home/arai/.mozbuild/sysroot-wasm32-wasi.tar.zst"
 0:23.71 checking for the wasm C compiler... /home/arai/.mozbuild/clang/bin/clang
 0:23.74 checking whether the wasm C compiler can be used... yes
 0:23.74 checking the wasm C compiler version... 13.0.0
 0:23.75 checking the wasm C compiler works... yes
 0:23.75 checking for the wasm C++ compiler... /home/arai/.mozbuild/clang/bin/clang++
 0:23.78 checking whether the wasm C++ compiler can be used... yes
 0:23.78 checking the wasm C++ compiler version... 13.0.0
 0:23.79 checking the wasm C++ compiler works... yes
 0:23.79 Installing bootstrapped toolchain in /home/arai/.mozbuild/dump_syms
 0:25.52 Setting up artifact dump_syms.tar.zst
 0:25.52 Downloading artifact to local cache: /home/arai/.mozbuild/toolchains/95284816ca2796ee-dump_syms.tar.zst
 0:25.66 Downloading... 0.0 %
 0:25.69 Downloading... 5.0 %
 0:25.73 Downloading... 10.1 %
 0:25.76 Downloading... 15.1 %
 0:25.76 Downloading... 20.1 %
 0:25.79 Downloading... 25.2 %
 0:25.81 Downloading... 30.2 %
 0:25.84 Downloading... 35.2 %
 0:25.86 Downloading... 40.2 %
 0:25.89 Downloading... 45.3 %
 0:25.91 Downloading... 50.0 %
 0:25.93 Downloading... 55.0 %
 0:25.96 Downloading... 60.1 %
 0:25.98 Downloading... 65.1 %
 0:26.01 Downloading... 70.1 %
 0:26.03 Downloading... 75.2 %
 0:26.06 Downloading... 80.2 %
 0:26.08 Downloading... 85.2 %
 0:26.11 Downloading... 90.2 %
 0:26.13 Downloading... 95.3 %
 0:26.15 Downloading... 100.0 %
 0:26.17 untarring "/home/arai/.mozbuild/dump_syms.tar.zst"
 0:26.27 checking for dump_syms... /home/arai/.mozbuild/dump_syms/dump_syms

on pattern (c), I see the following output in ./mach build

 0:03.11 checking for nasm... /usr/bin/nasm
 0:03.11 checking nasm version... 2.15.05
 0:03.11 checking for dump_syms... not found
 0:03.14 checking for getcontext... yes
Assignee: nobody → ahochheiden

On mozilla-release, auto-bootstrap is not enabled by default. So if you haven't auto-bootstrapped once already from mozilla-central, you won't have everything in place, yielding that error. If you add --enable-bootstrap to your config options, the build will auto-bootstrap.

If you add --enable-bootstrap to your config options, the build will auto-bootstrap.

As I tested, it won't work.

$ cat src/mozilla-release/browser/config/mozconfig
# This Source Code Form is subject to the terms of the Mozilla Public
# License, v. 2.0. If a copy of the MPL was not distributed with this
# file, You can obtain one at http://mozilla.org/MPL/2.0/.

# This file specifies the build flags for Firefox.  You can use it by adding:
#  . $topsrcdir/browser/config/mozconfig
# to the top of your mozconfig file.

ac_add_options --enable-application=browser
ac_add_options --with-branding=browser/branding/official
ac_add_options --enable-bootstrap

$ ls -l mozbuild                     
total 40
drwxr-xr-x 2 100999 100999 4096 11月 25 03:50 cbindgen
drwxr-xr-x 7 100999 100999 4096 11月 25 03:39 clang
drwxr-xr-x 3 lei    lei    4096 12月  8 19:30 clang-tools
drwxr-xr-x 2 100999 100999 4096 11月 25 03:52 fix-stacks
drwxr-xr-x 2 100999 100999 4096 11月 25 03:51 minidump_stackwalk
drwxr-xr-x 2 100999 100999 4096 11月 25 03:45 nasm
drwxr-xr-x 6 100999 100999 4096 12月  8 14:49 node
drwxr-xr-x 2 100999 100999 4096 11月 25 03:51 sccache
drwxr-xr-x 2 lei    lei    4096 12月  8 19:31 toolchains
drwxr-xr-x 3 lei    lei    4096 12月  8 19:26 _virtualenvs

 0:08.21 checking for the Mozilla API key... no
 0:08.21 checking for the Google Location Service API key... no
 0:08.21 checking for the Google Safebrowsing API key... no
 0:08.21 checking for the Bing API key... no
 0:08.21 checking for the Adjust SDK key... no
 0:08.21 checking for the Leanplum SDK key... no
 0:08.21 checking for the Pocket API key... no
 0:08.22 checking for x11 xcb xcb-shm x11-xcb xext xrandr >= 1.4.0 xcomposite xcursor xdamage xfixes xi... yes
 0:08.23 checking MOZ_X11_CFLAGS...
 0:08.24 checking MOZ_X11_LIBS... -lxcb-shm -lX11-xcb -lX11 -lxcb -lXext -lXrandr -lXcomposite -lXcursor -lXdamage -lXfixes -lXi
 0:08.25 checking for ice sm... yes
 0:08.26 checking MOZ_X11_SM_CFLAGS...
 0:08.27 checking for nasm... /usr/bin/nasm
 0:08.28 checking nasm version... 2.14.02
 0:08.28 ERROR: Cannot find a wasi sysroot. Please give its location with --with-wasi-sysroot. Or build with --without-wasm-sandboxed-libraries.
Error running mach:

    ['build']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file build| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

Exception: Process executed with non-0 exit code 1: ['/usr/bin/python3', '/src/mozilla-release/configure.py']

  File "/src/mozilla-release/python/mozbuild/mozbuild/build_commands.py", line 220, in build
    return driver.build(
  File "/src/mozilla-release/python/mozbuild/mozbuild/controller/building.py", line 1145, in build
    config_rc = self.configure(
  File "/src/mozilla-release/python/mozbuild/mozbuild/controller/building.py", line 1526, in configure
    status = self._run_command_in_objdir(
  File "/src/mozilla-release/python/mozbuild/mozbuild/base.py", line 853, in _run_command_in_objdir
    return self.run_process(cwd=self.topobjdir, **args)
  File "/src/mozilla-release/python/mach/mach/mixin/process.py", line 174, in run_process
    raise Exception(

Jake, which revision were you on when you tested in the above comment?
The auto-bootstrapping feature has been evolving rapidly in recent months, so narrowing doing which "state of the code" that your behaviour is associated with will make it more helpful.
You can get the revision hash with hd id

Flags: needinfo?(dwng.jr)

I'm experiencing the same issue , hg id results in e3466f2d6b39 tip (first time set up within the last week).
Edit: in my case the issue was probably having hg up to an unspecified bookmark, maybe unrelated to the issue.

Jake, which revision were you on when you tested in the above comment?

# hg clone from mozilla-release
+ hg log -r tip
changeset:   671273:2830e3f3d9af
tag:         tip
user:        Mozilla Releng Treescript <release+treescript@mozilla.org>
date:        Tue Dec 07 13:49:05 2021 +0000
summary:     Automatic version bump CLOSED TREE NO BUG a=release DONTBUILD
Flags: needinfo?(dwng.jr)
$ hd id
d7ba6bbdc41b+

In a fresh Ubuntu 20..04 VM on EC2:

sudo apt update
sudo apt install mercurial python3-distutils python3-pip
hg clone https://hg.mozilla.org/mozilla-unified
cd mozilla-unified
hg up -r d7ba6bbdc41bf0c76d05937c5104894d98637d92
./mach bootstrap
echo ac_add_options --enable-bootstrap > mozconfig
./mach build

Worked.

Same error on Windows.

$ hg id
2830e3f3d9af tip

This is mozilla-release.

$ hg pull && hg update --clean
no changes found
0 files updated, 0 files merged, 0 files removed, 0 files unresolved

Thing is, I cannot do bootstrap (non-artifact mode), because bootstrap crashes with

Could not find artifacts for a toolchain build named win64-nasm.

Any ideas?

Can something that's not been cleared by hg up --clean interfere with the bootstraping process? I've run into this problem a couple of times but it was because of local modifications including untracked files left around (hg up --clean won't wipe those IIRC).

Rather than having bootstrap download the dependency, if we wanted to specify --with-wasi-sysroot in mozconfig, where can we manually pull down wasi-sysroot ? Is it the standard one that comes with WASI SDK which is available at https://github.com/WebAssembly/wasi-sdk/releases (but how do we know which wasi-sysroot version to use for a particular version of firefox) ?

Thanks!

So the second month where I cannot update my Firefox just started I guess. Still the same problem.

(In reply to garrona from comment #12)

So the second month where I cannot update my Firefox just started I guess. Still the same problem.

Try the following:

  1. Update to current mozilla-central
  2. Make sure there are no changes to the source
  3. Make sure you have ac_add_options --enable-bootstrap in your .mozconfig
  4. Run ./mach --no-interactive bootstrap --application browser --no-system-changes
  5. Run ./mach configure

The wasi-sysroot artifact should be downloaded during step 5. If one of the steps fails then there's something wrong and we should investigate. Note that this works for me on all platforms, not just Windows.

It should work on the release branch, per comment 8, not just Ubuntu (well, without apt on other platforms).

(In reply to Gabriele Svelto [:gsvelto] from comment #13)

  1. Make sure you have ac_add_options --enable-bootstrap in your .mozconfig
  2. Run ./mach configure

These two combined did it, the ordinary mach bootstrap did nothing. It used all the archives it downloaded already and was happy with that.

The above (3 & 5) then forced downloading of some more parts I seem to not have had besides wasi (but I guess did not need during the past) during the configure step.
But I don't understand, why does the normal mach bootstrap not download all this stuff? I thought this is what it is supposed to do?

Thank you, it's compiling now.

(In reply to garrona from comment #15)

The above (3 & 5) then forced downloading of some more parts I seem to not have had besides wasi (but I guess did not need during the past) during the configure step.
But I don't understand, why does the normal mach bootstrap not download all this stuff? I thought this is what it is supposed to do?

I thought that too, but I experienced the same behavior recently on my machine so I was wondering about it. Sounds like a bug.

Thank you, it's compiling now.

You're welcome!

I just ran into this error on Fedora 35 on latest mozilla-central and none of the above helped. Removing the object directory and rebuilding got things working again.

See Also: → 1752835

Quoting Mitch from 1752835 for context.

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #5)

I don't necessarily disagree, but (conveniently) the build team had a discussion about our options here last night.
The use cases we need to support:

  • Developers building on central: use bootstrapping
  • Developers building a specific release (you are here 😉): use bootstrapping
  • Firefox CI doing releases: use "bootstrapping" (implemented as "fetches" in CI, but it's the same idea)
  • Downstream packagers of Firefox: use system tools
  • Developers building a specific release to release as part of their distro (rare, but does happen): use system tools

Options:

  • [glandium] Assume "should bootstrap" if a VCS is involved - this is pretty implicit and a weak connection though, IMO
  • [glandium] Assume "should bootstrap" if ./mach is used instead of ./configure && make - however, this restricts our ability to move to different "build backends"
  • [mitch] Assume "should use bootstrapped tools" if ~/.mozbuild is populated (so, if someone had manually run ./mach bootstrap)
    • [glandium] this forces the developer doing distro release builds to specify --disable-bootstrap
      • [mitch] I think that's an OK tradeoff?
  • [glandium] Make ./mach bootstrap automatically create/update mozconfig to add --enable-bootstrap, [remove bootstrap inference]?
    • [mitch] This is blocked on mozconfig becoming declarative rather than an interpreted shell script, which isn't happening anytime soon

Ultimately, there is no silver bullet solution that can support all use cases. Somebody either has to opt-in (--enable-bootstrap) or opt-out (--disable-bootstrap), so changing the default behavior here to support this use case just shifts the burden of manually writing a mozconfig elsewhere, which doesn't really solve anything.

The default of if --enable-bootstrap is set or not is controlled by this code. To briefly summarize the current behavior: If you're on mozilla-central (aka nightly) and using a VCS (hg or git), --enable-bootstrap will be set by default. If you are in any other context (on mozilla-release, in CI, not using a VCS, etc.) --enable-bootstrap is not set and the build will attempt to use the system tools by default.

Workaround

Add the following to your mozconfig file:
ac_add_options --enable-bootstrap

Status: NEW → RESOLVED
Closed: 3 years ago
Resolution: --- → WONTFIX

With Firefox 98.0, despite ac_add_options --enable-bootstrap and

0:03.92 Adding configure options from /dev/shm/bee-pmenzel/firefox/firefox-98.0-0/source/mozconfig
 0:03.92   --prefix=/usr
 0:03.92   --enable-application=browser
 0:03.92   --disable-necko-wifi
 0:03.92   --enable-official-branding
 0:03.92   --enable-system-pixman
 0:03.92   --without-system-icu
 0:03.92   --without-system-nspr
 0:03.92   --disable-tests
 0:03.92   --enable-optimize
 0:03.92   --disable-crashreporter
 0:03.92   --disable-updater
 0:03.92   --enable-bootstrap

I am getting that error:

 0:22.12 checking nasm version... 2.14
 0:22.12 ERROR: Cannot find a wasi sysroot. Please give its location with --with-wasi-sysroot. Or build with --without-wasm-sandboxed-libraries.
Error running mach:

    ['build']

Paul, can you create a new ticket for your issue? I'm guessing that it's related, but a different problem.
Also, please share your $objdir/config.log.

Flags: needinfo?(pmenzel+bugzilla.mozilla.org)

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #20)

Paul, can you create a new ticket for your issue? I'm guessing that it's related, but a different problem.
Also, please share your $objdir/config.log.

Done: https://bugzilla.mozilla.org/show_bug.cgi?id=1759544

Flags: needinfo?(pmenzel+bugzilla.mozilla.org)
You need to log in before you can comment on or make changes to this bug.