1743832 - Bootstrap not installing fix-stacks

Status: RESOLVED FIXED

(Firefox Build System :: Bootstrap Configuration, defect)

Description

[Dave Townsend [:mossop]]

I got the following error during a test run:

Exception in thread ProcessReader:
Traceback (most recent call last):
  File "/Users/dave/dotfiles/external/pyenv/versions/3.9.9/lib/python3.9/threading.py", line 973, in _bootstrap_inner
    self.run()
  File "/Users/dave/mozilla/source/trunk/third_party/python/sentry_sdk/sentry_sdk/integrations/threading.py", line 69, in run
    reraise(*_capture_exception())
  File "/Users/dave/mozilla/source/trunk/third_party/python/sentry_sdk/sentry_sdk/_compat.py", line 57, in reraise
    raise value
  File "/Users/dave/mozilla/source/trunk/third_party/python/sentry_sdk/sentry_sdk/integrations/threading.py", line 67, in run
    return old_run_func(self, *a, **kw)
  File "/Users/dave/dotfiles/external/pyenv/versions/3.9.9/lib/python3.9/threading.py", line 910, in run
    self._target(*self._args, **self._kwargs)
  File "/Users/dave/mozilla/source/trunk/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1230, in _read
    callback(line.rstrip())
  File "/Users/dave/mozilla/source/trunk/testing/mozbase/mozprocess/mozprocess/processhandler.py", line 1136, in __call__
    e(*args, **kwargs)
  File "/Users/dave/mozilla/build/trunk/obj-browser-dbg-artifact/_tests/testing/mochitest/runtests.py", line 3401, in processOutputLine
    message = handler(message)
  File "/Users/dave/mozilla/build/trunk/obj-browser-dbg-artifact/_tests/testing/mochitest/runtests.py", line 3494, in fix_stack
    message["data"] = self.stackFixerFunction(message["data"])
  File "/Users/dave/mozilla/source/trunk/testing/mozbase/mozrunner/mozrunner/utils.py", line 289, in stack_fixer_function
    return stack_fixer_module.fixSymbols(
  File "/Users/dave/mozilla/build/trunk/obj-browser-dbg-artifact/dist/bin/fix_stacks.py", line 82, in fixSymbols
    initFixStacks(jsonMode, slowWarning, breakpadSymsDir, hide_errors)
  File "/Users/dave/mozilla/build/trunk/obj-browser-dbg-artifact/dist/bin/fix_stacks.py", line 38, in initFixStacks
    raise Exception("cannot find `fix-stacks`; please run `./mach bootstrap`")
Exception: cannot find `fix-stacks`; please run `./mach bootstrap`

But I have run ./mach bootstrap and when I just tried again it didn't install anything.

Comment 1

[Mitchell Hentges [:mhentges] 🦀]

A few questions:

1. What's the results of ls -l ~/.mozbuild/fix-stacks?
2. Can you chuck some print()s around here to find out what the value of fix_stacks_exe is?
3. What command are you running, specifically?

Comment 2

[Dave Townsend [:mossop]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #1)

A few questions:

1. What's the results of ls -l ~/.mozbuild/fix-stacks?

ls: /Users/dave/.mozbuild/fix-stacks: No such file or directory

2. Can you chuck some print()s around here to find out what the value of fix_stacks_exe is?

/Users/dave/.mozbuild/fix-stacks/fix-stacks

3. What command are you running, specifically?

mach test browser/components/places/

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

ls: /Users/dave/.mozbuild/fix-stacks: No such file or directory

Ok, this points to bootstrap as being the problem here.
Can you share the full output of ./mach bootstrap?

Comment 4

[Dave Townsend [:mossop]]

(In reply to Mitchell Hentges [:mhentges] 🦀 from comment #3)

ls: /Users/dave/.mozbuild/fix-stacks: No such file or directory

Ok, this points to bootstrap as being the problem here.
Can you share the full output of ./mach bootstrap?

trunk$ mach bootstrap
created virtual environment CPython3.9.9.final.0-64 in 17ms
  creator CPython3Posix(dest=/Users/dave/.mozbuild/srcdirs/trunk-d2ed815e765d/_virtualenvs/mach, clear=False, no_vcs_ignore=False, global=False)
  activators BashActivator,CShellActivator,FishActivator,PowerShellActivator,PythonActivator
Collecting glean-sdk==40.0.0
  Using cached glean_sdk-40.0.0-cp36-abi3-macosx_11_0_arm64.whl (833 kB)
Collecting cffi>=1.13.0
  Using cached cffi-1.15.0-cp39-cp39-macosx_11_0_arm64.whl (173 kB)
Collecting glean-parser==3.6.0
  Using cached glean_parser-3.6.0-py3-none-any.whl (80 kB)
Requirement already satisfied: diskcache>=4 in ./third_party/python/diskcache (from glean-parser==3.6.0->glean-sdk==40.0.0) (4.1.0)
Requirement already satisfied: PyYAML>=5.3.1 in ./third_party/python/PyYAML/lib3 (from glean-parser==3.6.0->glean-sdk==40.0.0) (5.4.1)
Requirement already satisfied: appdirs>=1.4 in ./third_party/python/appdirs (from glean-parser==3.6.0->glean-sdk==40.0.0) (1.4.4)
Requirement already satisfied: Jinja2>=2.10.1 in ./third_party/python/Jinja2 (from glean-parser==3.6.0->glean-sdk==40.0.0) (2.11.3)
Requirement already satisfied: yamllint>=1.18.0 in ./third_party/python/yamllint (from glean-parser==3.6.0->glean-sdk==40.0.0) (1.23.0)
Requirement already satisfied: Click>=7 in ./third_party/python/click (from glean-parser==3.6.0->glean-sdk==40.0.0) (7.1.2)
Requirement already satisfied: jsonschema<4,>=3.0.2 in ./third_party/python/jsonschema (from glean-parser==3.6.0->glean-sdk==40.0.0) (3.2.0)
Collecting pycparser
  Using cached pycparser-2.21-py2.py3-none-any.whl (118 kB)
Requirement already satisfied: MarkupSafe>=0.23 in ./third_party/python/MarkupSafe/src (from Jinja2>=2.10.1->glean-parser==3.6.0->glean-sdk==40.0.0) (1.1.1)
Requirement already satisfied: six>=1.11.0 in ./third_party/python/six (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (1.13.0)
Requirement already satisfied: setuptools in ./third_party/python/setuptools (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (51.2.0)
Requirement already satisfied: pyrsistent>=0.14.0 in ./third_party/python/pyrsistent (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (0.16.0)
Requirement already satisfied: attrs>=17.4.0 in ./third_party/python/attrs (from jsonschema<4,>=3.0.2->glean-parser==3.6.0->glean-sdk==40.0.0) (19.1.0)
Requirement already satisfied: pathspec>=0.5.3 in ./third_party/python/pathspec (from yamllint>=1.18.0->glean-parser==3.6.0->glean-sdk==40.0.0) (0.9.0)
Installing collected packages: pycparser, glean-parser, cffi, glean-sdk
Successfully installed cffi-1.15.0 glean-parser-3.6.0 glean-sdk-40.0.0 pycparser-2.21
Collecting psutil<=5.8.0,>=5.4.2
  Using cached psutil-5.8.0-cp39-cp39-macosx_12_0_arm64.whl
Installing collected packages: psutil
Successfully installed psutil-5.8.0
Collecting zstandard<=0.16.0,>=0.11.1
  Using cached zstandard-0.16.0-cp39-cp39-macosx_11_0_arm64.whl (489 kB)
Installing collected packages: zstandard
Successfully installed zstandard-0.16.0
Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://firefox-source-docs.mozilla.org/contributing/build/artifact_builds.html.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode [default]
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
  5. SpiderMonkey JavaScript engine
Your choice:2
Your version of Mercurial (6.0) is sufficiently modern.
Your version of Rust (1.56.1) is new enough.
Rust supports aarch64-apple-darwin targets.

Mozilla recommends a number of changes to Mercurial to enhance your
experience with it.

Would you like to run a configuration wizard to ensure Mercurial is
optimally configured? (Yn): n

Your system should be ready to build Firefox for Desktop!

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Rust supports aarch64-apple-darwin targets.

Oh, are you on an M1 Mac? Perhaps that's related.
:glandium, my knowledge of M1 support is lacking (we should talk about this in our next meeting) - I'd like to get your take here before I continue investigating.

Comment 6

[Markus Stange [:mstange]]

I ran into this today when running dmd.py, which also expects fix-stacks to be present in .mozbuild.

Comment 7

[Mitchell Hentges [:mhentges] 🦀]

Markus are you on an ARM64 Mac as well, or which platform do you use?

Comment 8

[Markus Stange [:mstange]]

Yes, I'm also on an M1 Mac.

Comment 9

[Mitchell Hentges [:mhentges] 🦀]

:glandium, my knowledge of M1 support is lacking (we should talk about this in our next meeting) - I'd like to get your take here before I continue investigating.

FWIW, we chatted about this a bit on Monday - M1 bootstrapping should be fully working these days, and cases like there are indeed bugs, and not so-far-unimplemented behaviour.

Anyways, that doesn't help for this bug itself.
I can't reproduce (or look into) this too closely because I don't have an M1 Mac (yet). I'll leave the :glandium NI up here for investigation.

(update: I'll have access to an M1 Mac in late-January to look into this, if it hasn't been resolved by then)

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1743832 - Move bootstrap-related configure code to a separate file.

Comment 11

[Mike Hommey [:glandium]]

Attachment: Bug 1743832 - Autobootstrap fix-stacks if it's not available when needed.

Comment 12

[Mike Hommey [:glandium]]

The patches allow tests to work, but note that fix-stacks doesn't actually work, so stack traces won't be very useful. Bug 1746558 will fix that part.

Comment 13

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/a9d4bb3d1487
Move bootstrap-related configure code to a separate file. r=firefox-build-system-reviewers,andi
https://hg.mozilla.org/integration/autoland/rev/b519dfd40c39
Autobootstrap fix-stacks if it's not available when needed. r=firefox-build-system-reviewers,andi

Comment 14

[Mike Hommey [:glandium]]

The patches allow tests to work, but note that fix-stacks doesn't actually work, so stack traces won't be very useful. Bug 1746558 will fix that part.

Comment 15

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/a9d4bb3d1487
https://hg.mozilla.org/mozilla-central/rev/b519dfd40c39