Closed Bug 1684915 Opened 5 years ago Closed 5 years ago

./mach bootstrap does not install python-devel before other dependencies

Categories

(Firefox Build System :: Bootstrap Configuration, defect, P3)

Product:
Firefox Build System
Component:
Bootstrap Configuration
Platform:
x86_64
Linux
Type:
defect

Tracking

(firefox86 fixed)

Status:
RESOLVED FIXED
Milestone:
86 Branch
Tracking Flags:
Tracking Status
firefox86 --- fixed

People

(Reporter: lth, Assigned: sheehan)

References

Details

(Keywords: good-first-bug)

Attachments

(1 file)

Bug 1684915: install Python development headers during bootstrap on Fedora/Debian flavors r?#build
48 bytes, text/x-phabricator-request
Details | Review

Fedora 33. Had a repeated ./mach bootstrap crash (compilation of zstandard) that was resolved by installing python-devel manually before bootstrapping.

Do you have the ./mach bootstrap crash handy? If so, would you mind pasting it in this bug?
Based on your description, I suspect that it's because Python.h couldn't be found.

To resolve this in bootstrap, you'll want to modify self.packages in centosfedora.py.

Flags: needinfo?(lhansen)

That was lucky, it was still in the backscroll :-) Yes, Python.h missing is indeed the problem:

[lhansen@yojimbo m-u]$ ./mach bootstrap

Note on Artifact Mode:

Artifact builds download prebuilt C++ components rather than building
them locally. Artifact builds are faster!

Artifact builds are recommended for people working on Firefox or
Firefox for Android frontends, or the GeckoView Java API. They are unsuitable
for those working on C++ code. For more information see:
https://developer.mozilla.org/en-US/docs/Artifact_builds.

Please choose the version of Firefox you want to build:
  1. Firefox for Desktop Artifact Mode
  2. Firefox for Desktop
  3. GeckoView/Firefox for Android Artifact Mode
  4. GeckoView/Firefox for Android
Your choice: 2
Your version of Python 3 (3.9.1) is new enough.
Your version of Python 2 (2.7.18) is new enough.
Executing as root: sudo dnf groupinstall "C Development Tools and Libraries"
Last metadata expiration check: 2:18:35 ago on Mon 04 Jan 2021 04:13:34 PM CET.
Dependencies resolved.
======================================================================================================================================
 Package                         Architecture                   Version                         Repository                       Size
======================================================================================================================================
Installing Groups:
 C Development Tools and Libraries
                                                                                                                                     

Transaction Summary
======================================================================================================================================

Is this ok [y/N]: y
Complete!
Executing as root: sudo dnf install nodejs which npm redhat-rpm-config
Last metadata expiration check: 2:18:39 ago on Mon 04 Jan 2021 04:13:34 PM CET.
Package nodejs-1:14.15.1-1.fc33.x86_64 is already installed.
Package which-2.21-20.fc33.x86_64 is already installed.
Package npm-1:6.14.8-1.14.15.1.1.fc33.x86_64 is already installed.
Package redhat-rpm-config-174-1.fc33.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
virtualenv at /home/lhansen/.mozbuild/_virtualenvs/mach is already up to date.
Requirement already satisfied: cffi==1.14.4 in /home/lhansen/.mozbuild/_virtualenvs/mach/lib64/python3.9/site-packages (from -r /home/lhansen/m-u/build/zstandard_requirements.txt (line 7)) (1.14.4)
Requirement already satisfied: pycparser==2.20 in /home/lhansen/.mozbuild/_virtualenvs/mach/lib/python3.9/site-packages (from -r /home/lhansen/m-u/build/zstandard_requirements.txt (line 45)) (2.20)
Collecting zstandard==0.13.0
  Using cached zstandard-0.13.0.tar.gz (659 kB)
Building wheels for collected packages: zstandard
  Building wheel for zstandard (setup.py) ... error
  ERROR: Command errored out with exit status 1:
   command: /home/lhansen/.mozbuild/_virtualenvs/mach/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"'; __file__='"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' bdist_wheel -d /tmp/pip-wheel-l87cfx7x
       cwd: /tmp/pip-install-15g3rak_/zstandard/
  Complete output (26 lines):
  generating build/_zstd_cffi.c
  (already up-to-date)
  not modified: 'build/_zstd_cffi.c'
  running bdist_wheel
  running build
  running build_py
  creating build/lib.linux-x86_64-3.9
  creating build/lib.linux-x86_64-3.9/zstandard
  copying zstandard/cffi.py -> build/lib.linux-x86_64-3.9/zstandard
  copying zstandard/__init__.py -> build/lib.linux-x86_64-3.9/zstandard
  running build_ext
  building 'zstd' extension
  creating build/temp.linux-x86_64-3.9
  creating build/temp.linux-x86_64-3.9/c-ext
  creating build/temp.linux-x86_64-3.9/zstd
  creating build/temp.linux-x86_64-3.9/zstd/common
  creating build/temp.linux-x86_64-3.9/zstd/compress
  creating build/temp.linux-x86_64-3.9/zstd/decompress
  creating build/temp.linux-x86_64-3.9/zstd/dictBuilder
  gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Izstd/common -Izstd/compress -Izstd/dictBuilder -Izstd/decompress -Ic-ext -Izstd -I/home/lhansen/.mozbuild/_virtualenvs/mach/include -I/usr/include/python3.9 -c c-ext/bufferutil.c -o build/temp.linux-x86_64-3.9/c-ext/bufferutil.o -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY= -fvisibility=hidden
  In file included from c-ext/bufferutil.c:9:
  c-ext/python-zstandard.h:10:10: fatal error: Python.h: No such file or directory
     10 | #include <Python.h>
        |          ^~~~~~~~~~
  compilation terminated.
  error: command '/usr/lib64/ccache/gcc' failed with exit code 1
  ----------------------------------------
  ERROR: Failed building wheel for zstandard
  Running setup.py clean for zstandard
Failed to build zstandard
DEPRECATION: Could not build wheels for zstandard which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
Installing collected packages: zstandard
    Running setup.py install for zstandard ... error
    ERROR: Command errored out with exit status 1:
     command: /home/lhansen/.mozbuild/_virtualenvs/mach/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"'; __file__='"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-u8rkckg5/install-record.txt --single-version-externally-managed --compile --install-headers /home/lhansen/.mozbuild/_virtualenvs/mach/include/site/python3.9/zstandard
         cwd: /tmp/pip-install-15g3rak_/zstandard/
    Complete output (26 lines):
    generating build/_zstd_cffi.c
    (already up-to-date)
    not modified: 'build/_zstd_cffi.c'
    running install
    running build
    running build_py
    creating build/lib.linux-x86_64-3.9
    creating build/lib.linux-x86_64-3.9/zstandard
    copying zstandard/cffi.py -> build/lib.linux-x86_64-3.9/zstandard
    copying zstandard/__init__.py -> build/lib.linux-x86_64-3.9/zstandard
    running build_ext
    building 'zstd' extension
    creating build/temp.linux-x86_64-3.9
    creating build/temp.linux-x86_64-3.9/c-ext
    creating build/temp.linux-x86_64-3.9/zstd
    creating build/temp.linux-x86_64-3.9/zstd/common
    creating build/temp.linux-x86_64-3.9/zstd/compress
    creating build/temp.linux-x86_64-3.9/zstd/decompress
    creating build/temp.linux-x86_64-3.9/zstd/dictBuilder
    gcc -pthread -Wno-unused-result -Wsign-compare -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DNDEBUG -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -O2 -fexceptions -g -grecord-gcc-switches -pipe -Wall -Werror=format-security -Wp,-D_FORTIFY_SOURCE=2 -Wp,-D_GLIBCXX_ASSERTIONS -fstack-protector-strong -m64 -mtune=generic -fasynchronous-unwind-tables -fstack-clash-protection -fcf-protection -D_GNU_SOURCE -fPIC -fwrapv -fPIC -Izstd -Ic-ext -Izstd/compress -Izstd/common -Izstd/dictBuilder -Izstd/decompress -I/home/lhansen/.mozbuild/_virtualenvs/mach/include -I/usr/include/python3.9 -c c-ext/bufferutil.c -o build/temp.linux-x86_64-3.9/c-ext/bufferutil.o -DZSTD_MULTITHREAD -DZSTDLIB_VISIBILITY= -DZDICTLIB_VISIBILITY= -DZSTDERRORLIB_VISIBILITY= -fvisibility=hidden
    In file included from c-ext/bufferutil.c:9:
    c-ext/python-zstandard.h:10:10: fatal error: Python.h: No such file or directory
       10 | #include <Python.h>
          |          ^~~~~~~~~~
    compilation terminated.
    error: command '/usr/lib64/ccache/gcc' failed with exit code 1
    ----------------------------------------
ERROR: Command errored out with exit status 1: /home/lhansen/.mozbuild/_virtualenvs/mach/bin/python -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"'; __file__='"'"'/tmp/pip-install-15g3rak_/zstandard/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-u8rkckg5/install-record.txt --single-version-externally-managed --compile --install-headers /home/lhansen/.mozbuild/_virtualenvs/mach/include/site/python3.9/zstandard Check the logs for full command output.
WARNING: You are using pip version 20.2.2; however, version 20.3.3 is available.
You should consider upgrading via the '/home/lhansen/.mozbuild/_virtualenvs/mach/bin/python -m pip install --upgrade pip' command.
Error running mach:

    ['create-mach-environment']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file create-mach-environment| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/home/lhansen/.mozbuild/_virtualenvs/mach/bin/pip', 'install', '--requirement', '/home/lhansen/m-u/build/zstandard_requirements.txt', '--require-hashes']' returned non-zero exit status 1.

  File "/home/lhansen/m-u/python/mozbuild/mozbuild/mach_commands.py", line 2098, in create_mach_environment
    manager.install_pip_requirements(
  File "/home/lhansen/m-u/python/mozbuild/mozbuild/virtualenv.py", line 657, in install_pip_requirements
    return self._run_pip(args)
  File "/home/lhansen/m-u/python/mozbuild/mozbuild/virtualenv.py", line 672, in _run_pip
    subprocess.check_call(
  File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
Error running mach:

    ['bootstrap']

The error occurred in code that was called by the mach command. This is either
a bug in the called code itself or in the way that mach is calling it.
You can invoke |./mach busted| to check if this issue is already on file. If it
isn't, please use |./mach busted file bootstrap| to report it. If |./mach busted| is
misbehaving, you can also inspect the dependencies of bug 1543241.

If filing a bug, please include the full output of mach, including this error
message.

The details of the failure are as follows:

subprocess.CalledProcessError: Command '['/usr/bin/python3', '/home/lhansen/m-u/mach', 'create-mach-environment']' returned non-zero exit status 1.

  File "/home/lhansen/m-u/python/mozboot/mozboot/mach_commands.py", line 58, in bootstrap
    bootstrapper.bootstrap()
  File "/home/lhansen/m-u/python/mozboot/mozboot/bootstrap.py", line 419, in bootstrap
    self.instance.ensure_mach_environment(checkout_root)
  File "/home/lhansen/m-u/python/mozboot/mozboot/base.py", line 290, in ensure_mach_environment
    subprocess.check_call(cmd, cwd=checkout_root)
  File "/usr/lib64/python3.9/subprocess.py", line 373, in check_call
    raise CalledProcessError(retcode, cmd)
Sentry is attempting to send 0 pending error messages
Waiting up to 2 seconds
Press Ctrl-C to quit
Flags: needinfo?(lhansen)
Keywords: good-first-bug
Priority: -- → P5

I have the same issue on a Debian (and therefor, cannot build Firefox). So, not sure it should be a P5... ;)

Installing python3-dev fixed the issue (not sure why I didn't have it and bootstrap doesn't recommend it anymore)

Ah, thought this Ubuntu/Debian installed libpython3-dev with python3 automatically. I'll take a look.

Assignee: nobody → mhentges
Status: NEW → ASSIGNED
Priority: P5 → P3
Assignee: mhentges → sheehan

Since zstandard has native code that must be compiled, and that code
uses Python headers, we should be installing those headers as part
of bootstrap.

Most users will have these packages on their machines through various
other means (notably installing pip, ie sudo apt install python3-pip),
but since it is possible to avoid a pip installation (for example
by installing Mercurial through yum and then running bootstrap
immediately after cloning) we should specify these packages as required
by bootstrap.

Pushed by cosheehan@mozilla.com: https://hg.mozilla.org/integration/autoland/rev/714f90ee2d8f install Python development headers during bootstrap on Fedora/Debian flavors r=firefox-build-system-reviewers,mhentges
Regressions: 1686333

https://hg.mozilla.org/mozilla-central/rev/714f90ee2d8f

Status: ASSIGNED → RESOLVED
Closed: 5 years ago
status-firefox86: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → 86 Branch
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: