1679616 - Perma psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory

Status: RESOLVED FIXED

(Firefox Build System :: General, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: ccoroiu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=322922117&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Py-83GPjQrOuwquoqSoMJA/runs/0/artifacts/public/logs/live_backing.log

---

Started to perma fail with this merge: 
https://treeherder.mozilla.org/jobs?repo=mozilla-central&revision=77486ce0e2bcf6f7b0beeef34dbadceb1cd4d3f6&searchStr=linux%2Cx64%2Cdebug%2Cstatic-analysis-autotest-linux64-st-autotest%2Fdebug%2Csa&selectedTaskRun=Py-83GPjQrOuwquoqSoMJA.0

Failure log: https://treeherder.mozilla.org/logviewer?job_id=322922117&repo=mozilla-central&lineNumber=770 

[task 2020-11-28T09:43:24.416Z]   copying psutil/tests/test_misc.py -> build/lib.linux-x86_64-3.7/psutil/tests
[task 2020-11-28T09:43:24.416Z]   copying psutil/tests/test_contracts.py -> build/lib.linux-x86_64-3.7/psutil/tests
[task 2020-11-28T09:43:24.416Z]   copying psutil/tests/test_process.py -> build/lib.linux-x86_64-3.7/psutil/tests
[task 2020-11-28T09:43:24.416Z]   copying psutil/tests/test_connections.py -> build/lib.linux-x86_64-3.7/psutil/tests
[task 2020-11-28T09:43:24.417Z]   copying psutil/tests/test_system.py -> build/lib.linux-x86_64-3.7/psutil/tests
[task 2020-11-28T09:43:24.418Z]   running build_ext
[task 2020-11-28T09:43:24.418Z]   building 'psutil._psutil_linux' extension
[task 2020-11-28T09:43:24.418Z]   creating build/temp.linux-x86_64-3.7
[task 2020-11-28T09:43:24.419Z]   creating build/temp.linux-x86_64-3.7/psutil
[task 2020-11-28T09:43:24.419Z]   x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -DPSUTIL_POSIX=1 -DPSUTIL_SIZEOF_PID_T=4 -DPSUTIL_VERSION=570 -DPSUTIL_LINUX=1 -I/builds/worker/workspace/obj-build/_virtualenvs/init_py3/include -I/usr/include/python3.7m -c psutil/_psutil_common.c -o build/temp.linux-x86_64-3.7/psutil/_psutil_common.o
[task 2020-11-28T09:43:24.420Z]   psutil/_psutil_common.c:9:10: fatal error: Python.h: No such file or directory
[task 2020-11-28T09:43:24.420Z]    #include <Python.h>
[task 2020-11-28T09:43:24.421Z]             ^~~~~~~~~~
[task 2020-11-28T09:43:24.421Z]   compilation terminated.
[task 2020-11-28T09:43:24.422Z]   error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
[task 2020-11-28T09:43:24.422Z]   ----------------------------------------
[task 2020-11-28T09:43:24.423Z]   ERROR: Failed building wheel for psutil
[task 2020-11-28T09:43:24.423Z]   Running setup.py clean for psutil
[task 2020-11-28T09:43:24.621Z] Failed to build psutil
[task 2020-11-28T09:43:24.622Z] DEPRECATION: Could not build wheels for psutil which do not use PEP 517. pip will fall back to legacy 'setup.py install' for these. pip 21.0 will remove support for this functionality. A possible replacement is to fix the wheel build issue reported above. You can find discussion regarding this at https://github.com/pypa/pip/issues/8368.
[task 2020-11-28T09:43:24.658Z]```

Comment 1

[Intermittent Failures Robot]

8 failures in 1479 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8

Platform breakdown:

• linux64: 8

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1679616&startday=2020-11-23&endday=2020-11-29&tree=all

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1666345

Comment 3

[Mitchell Hentges [:mhentges] 🦀]

Yep, the linked regression ticket checks out. I'll dig in today, thanks!

Comment 4

[Mitchell Hentges [:mhentges] 🦀]

Looks like it's because the one builder image that's failing is missing the python3-dev and python-dev packages.

Comment 5

[Mitchell Hentges [:mhentges] 🦀]

Attachment: Bug 1679616: Include python[3]-dev in static-analysis-build image

To build native python packages, the python dev packages are needed.
This should resolve the psutil installation failure.

Comment 6

[Daniel Holbert [:dholbert]]

I was running into this locally, because I was missing the python-dev package. After I installed it, I was able to complete ./mach bootstrap successfully.

For anyone else hitting this locally (who won't be helped by a CI build image update): you need to run:

apt-get install python-dev python3-dev

...or your linux distro's equivalent.

I suspect we need to add these packages to https://searchfox.org/mozilla-central/source/python/mozboot/mozboot/debian.py and similar (centosfedora.py, etc) in order for ./mach bootstrap to do what it's supposed to, probably...

Comment 7

[Daniel Holbert [:dholbert]]

I filed bug 1680162 on comment 6 (i.e. on fixing mozboot to avoid individual developers running into this, so that ./mach bootstrap continues to behave gracefully on a freshly installed machine, or on any machine that happens to lack these packages).

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/bace7f10b4f9
Include python[3]-dev in static-analysis-build image r=sylvestre

Comment 9

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/bace7f10b4f9

Comment 10

[Intermittent Failures Robot]

17 failures in 3712 pushes (0.005 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 17

Platform breakdown:

• linux64: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1679616&startday=2020-11-30&endday=2020-12-06&tree=all