1729760 - glsl-optimizer produces errors when building Gecko for Android

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[[:owlish] 🦉 PST]

Mac OS 11.4, non-artifact build, latest tip
The error from the ./mach build output:

2:35.73   running: "/Users/owlish/.mozbuild/clang/bin/clang++" "-O1" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-apple-darwin" "--sysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" "-std=gnu++17" "-DMOZILLA_CONFIG_H" "-I" "glsl-optimizer/include" "-I" "glsl-optimizer/src/mesa" "-I" "glsl-optimizer/src/mapi" "-I" "glsl-optimizer/src/compiler" "-I" "glsl-optimizer/src/compiler/glsl" "-I" "glsl-optimizer/src/gallium/auxiliary" "-I" "glsl-optimizer/src/gallium/include" "-I" "glsl-optimizer/src" "-I" "glsl-optimizer/src/util" "-D__STDC_FORMAT_MACROS" "-DHAVE_PTHREAD" "-DHAVE_TIMESPEC_GET" "-DMOZ_INCLUDE_MOZALLOC_H" "-Dmozilla_throw_gcc_h" "-o" "/Users/owlish/code/gecko/obj-x86_64-unknown-linux-android/release/build/glslopt-a5682f2bddabc4bc/out/glsl-optimizer/src/compiler/glsl_types.o" "-c" "glsl-optimizer/src/compiler/glsl_types.cpp"
 2:35.73   cargo:warning=clang-12: warning: include path for libstdc++ headers not found; pass '-stdlib=libc++' on the command line to use the libc++ standard library instead [-Wstdlibcxx-not-found]
 2:35.73   cargo:warning=In file included from glsl-optimizer/src/compiler/glsl_types.cpp:26:
 2:35.73   cargo:warning=In file included from glsl-optimizer/src/compiler/glsl/glsl_parser_extras.h:34:
 2:35.73   cargo:warning=glsl-optimizer/src/compiler/glsl/glsl_symbol_table.h:28:10: fatal error: 'new' file not found
 2:35.73   cargo:warning=#include <new>
 2:35.73   cargo:warning=         ^~~~~
 2:35.73   cargo:warning=1 error generated.
 2:35.73   exit status: 1
 2:35.73   --- stderr
 2:35.73   error occurred: Command "/Users/owlish/.mozbuild/clang/bin/clang++" "-O1" "-ffunction-sections" "-fdata-sections" "-fPIC" "--target=x86_64-apple-darwin" "--sysroot" "/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk" "-std=gnu++17" "-DMOZILLA_CONFIG_H" "-I" "glsl-optimizer/include" "-I" "glsl-optimizer/src/mesa" "-I" "glsl-optimizer/src/mapi" "-I" "glsl-optimizer/src/compiler" "-I" "glsl-optimizer/src/compiler/glsl" "-I" "glsl-optimizer/src/gallium/auxiliary" "-I" "glsl-optimizer/src/gallium/include" "-I" "glsl-optimizer/src" "-I" "glsl-optimizer/src/util" "-D__STDC_FORMAT_MACROS" "-DHAVE_PTHREAD" "-DHAVE_TIMESPEC_GET" "-DMOZ_INCLUDE_MOZALLOC_H" "-Dmozilla_throw_gcc_h" "-o" "/Users/owlish/code/gecko/obj-x86_64-unknown-linux-android/release/build/glslopt-a5682f2bddabc4bc/out/glsl-optimizer/src/compiler/glsl_types.o" "-c" "glsl-optimizer/src/compiler/glsl_types.cpp" with args "clang++" did not execute successfully (status code exit status: 1).
 2:35.73 warning: build failed, waiting for other jobs to finish...```

I tried clobber, bootstrap, deleting object directories, resetting emulator (just in case), specifying “-stdlib=libc++” in the CFLAGS and CXXFLAGS variables, setting various values in HOST_CC / HOST_CXX / CC / CXX variables, overriding CXX_x86_64_apple_darwin variable, updating xcode - nothing helped.

Finally, I tried working off of an older commit - a random one from August 23 and a random one from August 10 - on the latter I was able to build.

jrmuizel is also able to reproduce the error.

Comment 1

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

I saw this today too on a new objdir: I'm on f880d61153d3 from Aug 26.

Curiously, I have an existing objdir that is from a direct descendant of that commit (something I wrote) that still builds successfully.

At the end of my runs, I see:

25:38.75 make[1]: INTERNAL: Exiting with 17 jobserver tokens available; should be 16!

Maybe there's a race condition that explains this?

Comment 2

[[:owlish] 🦉 PST]

I have not experienced the message about jobserver tokens

Comment 3

[[:owlish] 🦉 PST]

If it helps, I am currently on 2d2223baff892bb9bc0a229a0b8546cc90127594 - that one builds for me

Comment 4

[Emily Toop (:fluffyemily)]

This is currently blocking at least 2 people on both the performance and GeckoView teams. We would appreciate it if this could be prioritised. Thanks.

Comment 5

[Jim Mathies [:jimm]]

Jamie, didn't you run into a problem with this recently?

Comment 6

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Looks like gfx will handle this, clearing NI. Feel free to ping again if needed.

Comment 7

[Jamie Nicol [:jnicol]]

I've not seen this before, no.

I'm on PTO so am unable to fix this quickly, and I don't have a Mac either.

A regression range would be very helpful from someone who is able to reproduce. Nothing has changed in glslopt recently so presumably something in our build system has.

Comment 8

[Bryce Seager van Dyk [:bryce] (he/him) - Not reading bugmail]

Moving component. Feel free to pass back if this is found to be a media issue.

Comment 9

[Jim Mathies [:jimm]]

Jeff, you ran into this at some point? Trying to understand what the cause is, and who would own fixing it.

Comment 10

[Brad Werth [:bradwerth]]

I will try to reproduce.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

FWIW, I don't know if there's anything graphics specific about this issue. This crate last changed Apr 22 2021 so unless it's been broken since then it was likely a change elsewhere that introduced the problem.

If anyone has an idea when this broke that would help narrowing down the cause.

Comment 12

[Jeff Muizelaar [:jrmuizel]]

owlish/Michael, do you know when this last worked?

Comment 13

[[:owlish] 🦉 PST]

Jeff, the commit that is working for me is this one https://hg.mozilla.org/mozilla-central/log?rev=99e249810495d252d478908c1e8333b6a3673c4e

I tried building on a commit dated August 23, and the build failed. So, apparently, something got merged between August 11 and August 23

Comment 14

[Jeff Muizelaar [:jrmuizel]]

Can you hg/git bisect the build to see what commit broke it?

Comment 15

[Jeff Muizelaar [:jrmuizel]]

My guess is this was caused by bug 1724374

Comment 16

[Jeff Muizelaar [:jrmuizel]]

Yeah, a local bisect seems to confirm that.

Comment 17

[Mike Hommey [:glandium]]

Setting HOST_CXXFLAGS=-stdlib=libc++ should work around this.

Comment 18

[[:owlish] 🦉 PST]

According to my git bisect it's this commit https://hg.mozilla.org/mozilla-central/rev/86d44d84b2cae5abd0df5af629e7c2827c592aa4
for this bug: https://bugzilla.mozilla.org/show_bug.cgi?id=1726628 - the previous commit builds for me

(I made a mistake about August 23 commits. Those do build for me - I might have confused commit shas there.)

mcomella, can you try git bisect as well? f880d61153d3 from Aug 26 does build for me - I wonder if there are two issues at play here

Comment 19

[Mike Hommey [:glandium]]

Attachment: Bug 1729760 - Set -stdlib=libc++ for mac builds at the toolchain level.

It's currently added manually to CXXFLAGS and bindgen flags, and is
notably missing from HOST_CXXFLAGS. However, setting it at the toolchain
level makes it inherited anywhere it's needed, including host builds and
bindgen.

Comment 20

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/88384fc94631
Set -stdlib=libc++ for mac builds at the toolchain level. r=firefox-build-system-reviewers,nalexander

Comment 21

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/88384fc94631

Comment 22

[[:owlish] 🦉 PST]

(In reply to Mike Hommey [:glandium] from comment #17)

Setting HOST_CXXFLAGS=-stdlib=libc++ should work around this.

For some reason I haven't noticed this message yesterday. I tried this, and it didn't help

Comment 23

[[:owlish] 🦉 PST]

But the latest tip builds for me, yay! Thanks everyone!

Comment 24

[Michael Comella (:mcomella) [NI reported issues only: ex-Mozilla]]

(In reply to [:owlish] 🦉 PST from comment #18)

mcomella, can you try git bisect as well? f880d61153d3 from Aug 26 does build for me - I wonder if there are two issues at play here

I'm assuming this isn't needed anymore. I'd tell you if it's working but I'm not actively building GV atm. I'll let you know if I run into trouble the next time I build.