1736502 - Perma clang-trunk CheckIncludeFile.c:1:10: fatal error: malloc/malloc.h: No such file or directory and other failures

Status: RESOLVED FIXED

(Firefox Build System :: Toolchains, defect, P5)

Description

[Treeherder Bug Filer]

Filed by: csabou [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=355236376&repo=mozilla-central
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/UynxIUZSRCaRcXEcj3dHqA/runs/0/artifacts/public/logs/live_backing.log

---

[task 2021-10-19T02:12:30.862Z] Contents of b'/builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeError.log' follow
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the include file malloc/malloc.h exists failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_bbc0d && [1/2] Building C object CMakeFiles/cmTC_bbc0d.dir/CheckIncludeFile.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: CMakeFiles/cmTC_bbc0d.dir/CheckIncludeFile.c.o 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] /usr/bin/gcc    -o CMakeFiles/cmTC_bbc0d.dir/CheckIncludeFile.c.o -c CheckIncludeFile.c
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] CheckIncludeFile.c:1:10: fatal error: malloc/malloc.h: No such file or directory
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]     1 | #include <malloc/malloc.h>
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]       |          ^~~~~~~~~~~~~~~~~
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] compilation terminated.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the include file valgrind/valgrind.h exists failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_89067 && [1/2] Building C object CMakeFiles/cmTC_89067.dir/CheckIncludeFile.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: CMakeFiles/cmTC_89067.dir/CheckIncludeFile.c.o 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] /usr/bin/gcc    -o CMakeFiles/cmTC_89067.dir/CheckIncludeFile.c.o -c CheckIncludeFile.c
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] CheckIncludeFile.c:1:10: fatal error: valgrind/valgrind.h: No such file or directory
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]     1 | #include <valgrind/valgrind.h>
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]       |          ^~~~~~~~~~~~~~~~~~~~~
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] compilation terminated.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the include file mach/mach.h exists failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_535bb && [1/2] Building C object CMakeFiles/cmTC_535bb.dir/CheckIncludeFile.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: CMakeFiles/cmTC_535bb.dir/CheckIncludeFile.c.o 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] /usr/bin/gcc    -o CMakeFiles/cmTC_535bb.dir/CheckIncludeFile.c.o -c CheckIncludeFile.c
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] CheckIncludeFile.c:1:10: fatal error: mach/mach.h: No such file or directory
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]     1 | #include <mach/mach.h>
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]       |          ^~~~~~~~~~~~~
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] compilation terminated.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the include file histedit.h exists failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_14991 && [1/2] Building C object CMakeFiles/cmTC_14991.dir/CheckIncludeFile.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: CMakeFiles/cmTC_14991.dir/CheckIncludeFile.c.o 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] /usr/bin/gcc    -o CMakeFiles/cmTC_14991.dir/CheckIncludeFile.c.o -c CheckIncludeFile.c
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] CheckIncludeFile.c:1:10: fatal error: histedit.h: No such file or directory
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]     1 | #include <histedit.h>
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]       |          ^~~~~~~~~~~~
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] compilation terminated.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the include file CrashReporterClient.h exists failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_de350 && [1/2] Building C object CMakeFiles/cmTC_de350.dir/CheckIncludeFile.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: CMakeFiles/cmTC_de350.dir/CheckIncludeFile.c.o 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] /usr/bin/gcc    -o CMakeFiles/cmTC_de350.dir/CheckIncludeFile.c.o -c CheckIncludeFile.c
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] CheckIncludeFile.c:1:10: fatal error: CrashReporterClient.h: No such file or directory
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]     1 | #include <CrashReporterClient.h>
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z]       |          ^~~~~~~~~~~~~~~~~~~~~~~
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] compilation terminated.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Determining if the function pfm_initialize exists in the pfm failed with the following output:
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] Run Build Command(s):/usr/bin/ninja cmTC_9744f && [1/2] Building C object CMakeFiles/cmTC_9744f.dir/CheckFunctionExists.c.o
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] [2/2] Linking C executable cmTC_9744f
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.862Z] FAILED: cmTC_9744f 
[task 2021-10-19T02:12:30.862Z] 
[task 2021-10-19T02:12:30.863Z] : && /usr/bin/gcc -DCHECK_FUNCTION_EXISTS=pfm_initialize -Wl,-Bsymbolic-functions -fuse-ld=gold -Wl,--gc-sections -Wl,--icf=safe CMakeFiles/cmTC_9744f.dir/CheckFunctionExists.c.o -o cmTC_9744f  -lpfm  -lm && :
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] /usr/bin/ld.gold: error: cannot find -lpfm
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] CMakeFiles/cmTC_9744f.dir/CheckFunctionExists.c.o:CheckFunctionExists.c:function main: error: undefined reference to 'pfm_initialize'
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] collect2: error: ld returned 1 exit status
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] ninja: build stopped: subcommand failed.
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] Performing C SOURCE FILE Test CMAKE_HAVE_LIBC_PTHREAD failed with the following output:
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] Change Dir: /builds/worker/fetches/llvm-project/build/stage1/build/CMakeFiles/CMakeTmp
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] Run Build Command(s):/usr/bin/ninja cmTC_d370d && [1/2] Building C object CMakeFiles/cmTC_d370d.dir/src.c.o
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] [2/2] Linking C executable cmTC_d370d
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] FAILED: cmTC_d370d 
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] : && /usr/bin/gcc -DCMAKE_HAVE_LIBC_PTHREAD -Wl,-Bsymbolic-functions -fuse-ld=gold -Wl,--gc-sections -Wl,--icf=safe CMakeFiles/cmTC_d370d.dir/src.c.o -o cmTC_d370d  -lm && :
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] CMakeFiles/cmTC_d370d.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_create'
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] CMakeFiles/cmTC_d370d.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_detach'
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] CMakeFiles/cmTC_d370d.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_cancel'
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] CMakeFiles/cmTC_d370d.dir/src.c.o:src.c:function main: error: undefined reference to 'pthread_join'
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] collect2: error: ld returned 1 exit status
[task 2021-10-19T02:12:30.863Z] 
[task 2021-10-19T02:12:30.863Z] ninja: build stopped: subcommand failed.

Comment 1

[Cosmin Sabou [:CosminS]]

Looks like this is not push related as it fails on older pushes like this one where it was manually added. Can't run it on a push where it was green because I get this error in the console Task `Bqklzth9QV6Pe838e60tBA` Can't be rescheduled past it's deadline of 2021-10-18T02:02:30.029Z.
It's not worker related because it fails on several ones:

• https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-3/worker-types/b-linux-xlarge/workers/us-west-1/i-0c34f2c551be1852a
• https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-3/worker-types/b-linux-xlarge/workers/us-west-2/i-0a9f292326f50d27f
• https://firefox-ci-tc.services.mozilla.com/provisioners/gecko-3/worker-types/b-linux-xlarge/workers/us-east-1/i-0fc6ee243966ae6bb

It looks more towards an infra error or with the setup of the job itself. First failed here.

Andi, please have a look over these failures. Thank you.

Comment 2

[Mike Hommey [:glandium]]

The real error is earlier:

[task 2021-10-19T02:12:28.814Z] CMake Error at projects/compiler-rt/cmake/Modules/AddCompilerRT.cmake:3 (include):
[task 2021-10-19T02:12:28.814Z]   include could not find load file:
[task 2021-10-19T02:12:28.814Z] 
[task 2021-10-19T02:12:28.814Z]     /builds/worker/fetches/llvm-project/llvm/projects/compiler-rt/../cmake/Modules/HandleCompilerRT.cmake
[task 2021-10-19T02:12:28.814Z] Call Stack (most recent call first):
[task 2021-10-19T02:12:28.814Z]   projects/compiler-rt/lib/CMakeLists.txt:4 (include)

This is probably caused by these upstream changes: https://reviews.llvm.org/rG0eed292fbae22a8856682b07e1cb968424b49941
We're victim of our continued use of the pre-monorepo ways of setting things up.

Comment 3

[Andi [:andi]]

This has been fixed, the issue from LLVM that glandium mentioned has been reverted.
https://treeherder.mozilla.org/jobs?repo=mozilla-central&searchStr=clang-trunk&revision=5a0ebbb0210fda2defe5a44a5b75f6f2488fe3c6

Comment 4

[Intermittent Failures Robot]

10 failures in 3186 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 8
• autoland: 1
• try: 1

Platform and build breakdown:

• toolchains: 10
  • opt: 10

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-10-18&endday=2021-10-24&tree=all

Comment 5

[Alexandru Michis [:malexandru]]

Andi, this issue has appeared again, looks to be the same issue.
Could you please take a look?

Comment 6

[Intermittent Failures Robot]

7 failures in 2845 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• toolchains: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-10-25&endday=2021-10-31&tree=all

Comment 7

[Andi [:andi]]

I'm discussion with the author of the patch that caused the regression to see what we can do to fix it.

Comment 8

[Intermittent Failures Robot]

7 failures in 3030 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 7

Platform and build breakdown:

• toolchains: 7
  • opt: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-11-01&endday=2021-11-07&tree=all

Comment 9

[Intermittent Failures Robot]

9 failures in 2723 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 2
• mozilla-central: 7

Platform and build breakdown:

• toolchains: 9
  • opt: 9

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-11-08&endday=2021-11-14&tree=all

Comment 10

[Mike Hommey [:glandium]]

Attachment: Bug 1736502 - Work around clang trunk changes that break our clang-trunk build.

Short of solving this by fully embracing the monorepo build and removing
the projects/ workarounds (which will also require a compat more with
older clang versions we still build with the same script), we can solve
the bustage with a small hack.

Comment 11

[Mike Hommey [:glandium]]

Note this yields, so upstream changes since the bustage have busted the clang-plugin:

error: 'error' diagnostics expected but not seen:
  File /builds/worker/checkouts/gecko/build/clang-plugin/tests/TestGlobalClass.cpp Line 19: variable of type 'Global [2]' only valid as global
error: 'error' diagnostics seen but not expected:
  File /builds/worker/checkouts/gecko/build/clang-plugin/tests/TestGlobalClass.cpp Line 19: variable of type 'Global[2]' only valid as global
error: 'note' diagnostics expected but not seen:
  File /builds/worker/checkouts/gecko/build/clang-plugin/tests/TestGlobalClass.cpp Line 19: 'Global [2]' is a global type because it is an array of global type 'Global'
error: 'note' diagnostics seen but not expected:
  File /builds/worker/checkouts/gecko/build/clang-plugin/tests/TestGlobalClass.cpp Line 19: 'Global[2]' is a global type because it is an array of global type 'Global'
4 errors generated.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/c8b2765f771c
Work around clang trunk changes that break our clang-trunk build. r=andi

Comment 13

[Narcis Beleuzu [:NarcisB]]

https://hg.mozilla.org/mozilla-central/rev/c8b2765f771c

Comment 14

[Intermittent Failures Robot]

4 failures in 3259 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 4

Platform and build breakdown:

• toolchains: 4
  • opt: 4

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-11-15&endday=2021-11-21&tree=all

Comment 15

[Cristian Tuns]

Andi, this issue has appeared again, looks to be the same issue.
Could you please take a look?

• Push

Comment 16

[Intermittent Failures Robot]

17 failures in 1011 pushes (0.017 failures/push) were associated with this bug in the last 7 days.

This is the #38 most frequent failure this week.

Repository breakdown:

• autoland: 11
• mozilla-central: 6

Platform and build breakdown:

• toolchains: 17
  • opt: 17

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1736502&startday=2021-12-27&endday=2022-01-02&tree=all

Comment 17

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

Mike, could you take care of this?

Comment 18

[Mike Hommey [:glandium]]

Please file a new bug because it's not the same thing. (how come it's been flagged on autoland too? Backfilling these jobs doesn't do anything useful)