1496270 - Building geckodriver for Firefox 60ESR fails with: failed to run custom build command for `bzip2-sys v0.1.6`

Status: RESOLVED FIXED

(Testing :: geckodriver, defect, P2)

Description

[Thomas Deutschmann]

Attachment: build.log

Building Firefox 60.2.1 or 60.2.2 with "--enable-geckodriver" using Clang or GCC is failing with:

> 42:37.22 error: failed to run custom build command for `bzip2-sys v0.1.6`
> 42:37.22 process didn't exit successfully: `/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./release/build/bzip2-sys-6f8d2d214145b4a9/build-script-build` (exit code: 101)
> 42:37.22 --- stdout
> 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu")
> 42:37.22 OPT_LEVEL = Some("2")
> 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu")
> 42:37.22 HOST = Some("x86_64-unknown-linux-gnu")
> 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu")
> 42:37.22 TARGET = Some("x86_64-unknown-linux-gnu")
> 42:37.22 HOST = Some("x86_64-unknown-linux-gnu")
> 42:37.22 CC_x86_64-unknown-linux-gnu = None
> 42:37.22 CC_x86_64_unknown_linux_gnu = None
> 42:37.22 HOST_CC = Some("/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99")
> 42:37.23 TARGET = Some("x86_64-unknown-linux-gnu")
> 42:37.23 HOST = Some("x86_64-unknown-linux-gnu")
> 42:37.23 CFLAGS_x86_64-unknown-linux-gnu = None
> 42:37.23 CFLAGS_x86_64_unknown_linux_gnu = None
> 42:37.23 HOST_CFLAGS = None
> 42:37.23 CFLAGS = Some("-pipe -march=ivybridge -mtune=ivybridge -mno-xsaveopt -Wno-error=missing-prototypes -Wno-error=enum-compare -Wno-error=unused-function -Wno-error=deprecated-declarations -frecord-gcc-switches")
> 42:37.23 DEBUG = Some("false")
> 42:37.23 running: "/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-pipe" "-march=ivybridge" "-mtune=ivybridge" "-mno-xsaveopt" "-Wno-error=missing-prototypes" "-Wno-error=enum-compare" "-Wno-error=unused-function" "-Wno-error=deprecated-declarations" "-frecord-gcc-switches" "-m64" "-I" "bzip2-1.0.6" "-DBZ_NO_STDIO" "-o" "/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./x86_64-unknown-linux-gnu/release/build/bzip2-sys-17a1b1ab7e70b8a0/out/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"
> 42:37.23 
> 42:37.23 --- stderr
> 42:37.23 thread 'main' panicked at '
> 42:37.23 
> 42:37.23 Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-gcc -std=gnu99` installed?
> 42:37.23 
> 42:37.23 ', third_party/rust/cc/src/lib.rs:1984:5
> 42:37.23 stack backtrace:
> 42:37.23    0:     0x55c4612301de - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::h9aa5ffc216cd8bab
> 42:37.23    1:     0x55c461222106 - std::sys_common::backtrace::print::h20333f3cf15259f2
> 42:37.23    2:     0x55c4612200dd - std::panicking::default_hook::{{closure}}::hd1cdeeefd64cf4be
> 42:37.24    3:     0x55c46121fe50 - std::panicking::default_hook::h386fdb6b44148c45
> 42:37.24    4:     0x55c46122072c - std::panicking::rust_panic_with_hook::h655ee35696268a2c
> 42:37.24    5:     0x55c4612204d9 - std::panicking::continue_panic_fmt::h9e6d68e1d0b2da5c
> 42:37.24    6:     0x55c46122043d - std::panicking::begin_panic_fmt::h8726b3c9c9f319ff
> 42:37.24    7:     0x55c461219ca1 - cc::fail::h9d6d9599e20fc10f
> 42:37.24    8:     0x55c46120f07e - cc::Build::compile::h8192787e03a59719
> 42:37.24    9:     0x55c4612047e9 - build_script_build::main::h998515e8f8daa082
> 42:37.24   10:     0x55c461203c12 - std::rt::lang_start::{{closure}}::h44498c5e97f8754d
> 42:37.24   11:     0x55c4612201e2 - std::panicking::try::do_call::hf328099dc8204c6c
> 42:37.25   12:     0x55c46123cc69 - __rust_maybe_catch_panic
> 42:37.25   13:     0x55c46121ad95 - std::rt::lang_start_internal::h3d854016d73be659
> 42:37.26   14:     0x55c461204963 - main
> 42:37.26   15:     0x7fb6246c050a - __libc_start_main
> 42:37.26                                at ../csu/libc-start.c:308
> 42:37.26   16:     0x55c4612038f9 - _start
> 42:37.26   17:                0x0 - <unknown>
> 42:37.26 
> 42:37.26 warning: build failed, waiting for other jobs to finish...

We can build geckodriver in 62.x branch (i.e. 62.0.2 or 62.0.3).


> # gcc -v
> Using built-in specs.
> COLLECT_GCC=gcc
> COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-pc-linux-gnu/8.2.0/lto-wrapper
> Target: x86_64-pc-linux-gnu
> Configured with: /var/tmp/portage/sys-devel/gcc-8.2.0-r3/work/gcc-8.2.0/configure --host=x86_64-pc-linux-gnu --build=x86_64-pc-linux-gnu --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/gcc-bin/8.2.0 --includedir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include --datadir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0 --mandir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/man --infodir=/usr/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/info --with-gxx-include-dir=/usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0/include/g++-v8 --with-python-dir=/share/gcc-data/x86_64-pc-linux-gnu/8.2.0/python --enable-languages=c,c++,fortran --enable-obsolete --enable-secureplt --disable-werror --with-system-zlib --enable-nls --without-included-gettext --enable-checking=release --with-bugurl=https://bugs.gentoo.org/ --with-pkgversion='Gentoo 8.2.0-r3 p1.3' --disable-esp --enable-libstdcxx-time --enable-shared --enable-threads=posix --enable-__cxa_atexit --enable-clocale=gnu --enable-multilib --with-multilib-list=m32,m64 --disable-altivec --disable-fixed-point --enable-targets=all --enable-libgomp --disable-libmudflap --disable-libssp --disable-libmpx --disable-systemtap --enable-vtable-verify --enable-libvtv --enable-lto --without-isl --enable-libsanitizer --enable-default-pie --enable-default-ssp
> Thread model: posix
> gcc version 8.2.0 (Gentoo 8.2.0-r3 p1.3)

> # clang -v
> clang version 7.0.0 (tags/RELEASE_700/final)
> Target: x86_64-pc-linux-gnu
> Thread model: posix
> InstalledDir: /usr/lib/llvm/7/bin
> Selected GCC installation: /usr/lib/gcc/x86_64-pc-linux-gnu/8.2.0
> Candidate multilib: .;@m64
> Candidate multilib: 32;@m32
> Selected multilib: .;@m64

Comment 1

[Henrik Skupin [:whimboo][⌚️UTC+1]]

So the failure message is:

> Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-gcc -std=gnu99` installed?

It looks like it searches for `x86_64-pc-linux-gnu-gcc` but `/usr/lib/gcc/x86_64-pc-linux-gnu` is installed? 

I don't know about the package ecosystem on Gentoo, but does a symlink here maybe help?

Comment 2

[Thomas Deutschmann]

No, a missing symlink isn't the problem. See the failure when using Clang:

> 49:11.30 error: failed to run custom build command for `bzip2-sys v0.1.6`
> 49:11.30 process didn't exit successfully: `/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./release/build/bzip2-sys-545eaa606c062b5d/build-script-build` (exit code: 101)
> 49:11.30 --- stdout
> 49:11.30 TARGET = Some("x86_64-unknown-linux-gnu")
> 49:11.30 OPT_LEVEL = Some("2")
> 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu")
> 49:11.31 HOST = Some("x86_64-unknown-linux-gnu")
> 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu")
> 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu")
> 49:11.31 HOST = Some("x86_64-unknown-linux-gnu")
> 49:11.31 CC_x86_64-unknown-linux-gnu = None
> 49:11.31 CC_x86_64_unknown_linux_gnu = None
> 49:11.31 HOST_CC = Some("/usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang -std=gnu99")
> 49:11.31 TARGET = Some("x86_64-unknown-linux-gnu")
> 49:11.32 HOST = Some("x86_64-unknown-linux-gnu")
> 49:11.32 CFLAGS_x86_64-unknown-linux-gnu = None
> 49:11.32 CFLAGS_x86_64_unknown_linux_gnu = None
> 49:11.32 HOST_CFLAGS = None
> 49:11.32 CFLAGS = Some("-pipe -march=ivybridge -mtune=ivybridge -mno-xsaveopt -Wno-error=missing-prototypes -Wno-error=enum-compare -Wno-error=unused-function -Wno-error=deprecated-declarations")
> 49:11.32 DEBUG = Some("false")
> 49:11.32 running: "/usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang -std=gnu99" "-O2" "-ffunction-sections" "-fdata-sections" "-fPIC" "-pipe" "-march=ivybridge" "-mtune=ivybridge" "-mno-xsaveopt" "-Wno-error=missing-prototypes" "-Wno-error=enum-compare" "-Wno-error=unused-function" "-Wno-error=deprecated-declarations" "--target=x86_64-unknown-linux-gnu" "-I" "bzip2-1.0.6" "-DBZ_NO_STDIO" "-o" "/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver/./x86_64-unknown-linux-gnu/release/build/bzip2-sys-cdca6cdfca0915a9/out/bzip2-1.0.6/blocksort.o" "-c" "bzip2-1.0.6/blocksort.c"
> 49:11.32
> 49:11.33 --- stderr
> 49:11.33 thread 'main' panicked at '
> 49:11.33
> 49:11.33 Internal error occurred: Failed to find tool. Is `x86_64-pc-linux-gnu-clang -std=gnu99` installed?
> 49:11.33
> 49:11.34 ', third_party/rust/cc/src/lib.rs:1984:5
> 49:11.34 stack backtrace:
> 49:11.34    0:     0x558a4a6407ae - std::sys::unix::backtrace::tracing::imp::unwind_backtrace::he72067c4756fc64f
> 49:11.34                                at libstd/sys/unix/backtrace/tracing/gcc_s.rs:49
> 49:11.34    1:     0x558a4a631056 - std::sys_common::backtrace::print::h88c229e1b073f122
> 49:11.34                                at libstd/sys_common/backtrace.rs:71
> 49:11.34                                at libstd/sys_common/backtrace.rs:59
> 49:11.34    2:     0x558a4a62e4fd - std::panicking::default_hook::{{closure}}::hb998174544b4cbff
> 49:11.35                                at libstd/panicking.rs:211
> 49:11.36    3:     0x558a4a62e270 - std::panicking::default_hook::h2a69458a7b4a8d95
> 49:11.36                                at libstd/panicking.rs:227
> 49:11.36    4:     0x558a4a62eb1c - std::panicking::rust_panic_with_hook::h1b0bf5031744c801
> 49:11.36                                at libstd/panicking.rs:475
> 49:11.36    5:     0x558a4a62e719 - std::panicking::continue_panic_fmt::h46dd3d92ccb20144
> 49:11.36                                at libstd/panicking.rs:390
> 49:11.36    6:     0x558a4a62e67d - std::panicking::begin_panic_fmt::hc0cc09bb1a210514
> 49:11.36                                at libstd/panicking.rs:345
> 49:11.36    7:     0x558a4a62a5c1 - cc::fail::h4aba75b28ac954d2
> 49:11.37    8:     0x558a4a61ee5e - cc::Build::compile::h6369b354c0cac3a6
> 49:11.37    9:     0x558a4a613f63 - build_script_build::main::hc422d7e31efa21f1
> 49:11.37   10:     0x558a4a613332 - std::rt::lang_start::{{closure}}::h6819c502d40c00db
> 49:11.37   11:     0x558a4a62e602 - std::panicking::try::do_call::hdbb9d33e960bebf1
> 49:11.37                                at libstd/rt.rs:59
> 49:11.37                                at libstd/panicking.rs:310
> 49:11.37   12:     0x558a4a64d209 - __rust_maybe_catch_panic
> 49:11.37                                at libpanic_unwind/lib.rs:105
> 49:11.38   13:     0x558a4a6301c5 - std::rt::lang_start_internal::h4fca12a3897276e1
> 49:11.38                                at libstd/panicking.rs:289
> 49:11.38                                at libstd/panic.rs:392
> 49:11.38                                at libstd/rt.rs:58
> 49:11.38   14:     0x558a4a6140c3 - main
> 49:11.38   15:     0x7ff096d5f50a - __libc_start_main
> 49:11.38                                at ../csu/libc-start.c:308
> 49:11.38   16:     0x558a4a613029 - _start
> 49:11.38   17:                0x0 - <unknown>
> 49:11.39
> 49:11.39 gmake[4]: *** [/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/config/rules.mk:1017: force-cargo-program-build] Error 101
> 49:11.39 gmake[4]: Leaving directory '/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/ff/testing/geckodriver'
> 49:11.39 gmake[3]: *** [/var/tmp/portage/www-client/firefox-60.2.2/work/firefox-60.2.2/config/recurse.mk:73: testing/geckodriver/target] Error 2
> 49:11.39 gmake[3]: *** Waiting for unfinished jobs....


/usr/lib/llvm/7/bin/x86_64-pc-linux-gnu-clang exists (like the gcc counterpart).

And if there would be a missing symlink or something like that, wouldn't this affect other 'rust packages'? But firefox is built successfully with geckodriver disabled.


See https://bugzilla.mozilla.org/show_bug.cgi?id=1367995 (and https://github.com/alexcrichton/bzip2-rs/issues/30). Same issue but affecting "Windows".

Comment 3

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Lets see what Ted thinks about this problem. Here has way more knowledge than me when it comes to build steps with Rust.

Comment 4

[(not currently active) Ted Mielczarek]

I fixed this in bug 1409276 which landed in the Firefox 63 timeframe. If you look at your failing log there you'll see:
> 42:37.22 HOST_CC = Some("/usr/bin/x86_64-pc-linux-gnu-gcc -std=gnu99")

You're presumably setting `HOST_CC` to that string, and the Rust cc crate is trying to find a binary with that name (including the space). This was fixed upstream in the cc crate, but the vendored version in ESR60 doesn't contain that fix:
https://github.com/alexcrichton/cc-rs/commit/0e405f9e62ce1cb8132002febc55c1e31788c31d#diff-b4aea3e418ccdb71239b96952d9cddb6

Comment 5

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Oh, shall we then try to vendor in the new version of the cc crate on the esr60 branch?

Comment 6

[(not currently active) Ted Mielczarek]

We could do that, it shouldn't be harmful. You'd need to run `cargo update -p cc && ./mach vendor rust`.

Comment 7

[Thomas Deutschmann]

Thanks Ted, Upstream commit helped me to find https://bugzilla.mozilla.org/show_bug.cgi?id=1445528. For Gentoo I will just backport changes from "third_party/rust/cc/src/lib.rs" file and update ".cargo-checksum.json". First test run with Clang was successful.

Henrik, I would appreciate if you could update so that we can drop our (Gentoo) patch with next ESR update and use a complete updated cc package.

Comment 8

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Attachment: Bug 1496270 - Upgrade cc crate to 1.0.25. r?ted

Comment 9

[Henrik Skupin [:whimboo][⌚️UTC+1]]

Comment on attachment 9018182 [details]
Bug 1496270 - Upgrade cc crate to 1.0.25. r?ted

[ESR Uplift Approval Request]

If this is not a sec:{high,crit} bug, please state case for ESR consideration: The cc crate had a bug where it would ignore compiler arguments in compilers specified in environment variables. This is basically a backport of https://hg.mozilla.org/integration/mozilla-inbound/rev/ee4bd7464329a1b284d484554e22fdc1ee6fb954.

User impact if declined: None, only distributors and developers are affected

Fix Landed on Version: 63

Risk to taking this patch: Low

Why is the change risky/not risky? (and alternatives if risky): Failures only happen during compile time

String or UUID changes made by this patch: None

Comment 10

[Ryan VanderMeulen [:RyanVM]]

I can approve and land this now, but note that it won't be included in the tagged 60.3.0 release (that's already built and ready to ship on Tuesday). It'll be available if you build directly from ESR60 tip or for the next tagged release (whatever that is).

Comment 11

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9018182 [details]
Bug 1496270 - Upgrade cc crate to 1.0.25. r?ted

Fixes NPOTB code causing build bustage for downstream users. Approved for ESR60.

Comment 12

[Ryan VanderMeulen [:RyanVM]]

https://hg.mozilla.org/releases/mozilla-esr60/rev/048288040aac