1525760 - Add clang-based base toolchain jobs

Status: RESOLVED FIXED

(Firefox Build System :: Task Configuration, task)

Description

[Mike Hommey [:glandium]]

We now default to building with clang, and we have a declared minimum supported version, but we don't actually check that version works. We should.

Comment 1

[Mike Hommey [:glandium]]

Attachment: Bug 1525760 - Add a clang-based base toolchains task.

These are copies of the corresponding gcc-based base toolchain tasks,
with FORCE_GCC and the gcc dependency removed.

We also tweak things a little for those builds to actually end up green.

Comment 2

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/439db957d3e2
Add a clang-based base toolchains task. r=froydnj

Comment 3

[Pulsebot]

Backout by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/6d6ab46f6428
Backed out changeset 439db957d3e2 for linux64 toolchains bustages in toolkit/library/gtest/target CLOSED TREE

Comment 4

[Raul Gurzau (:RaulG)]

Backed out changeset 439db957d3e2 (bug 1525760) for linux64 toolchains bustages in toolkit/library/gtest/target CLOSED TREE

Backout link: https://hg.mozilla.org/integration/autoland/rev/6d6ab46f64283ed228ebe32cd5b67f614fb26cca

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&revision=439db957d3e21f3fcaa2cea6b08dd7eae93c0eaf&selectedJob=227340126

Log link: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=227340126&repo=autoland&lineNumber=36736

Log snippet:

[task 2019-02-09T03:58:43.893Z] 03:58:43 INFO - /builds/worker/workspace/build/src/sccache2/sccache /builds/worker/workspace/build/src/clang/bin/clang++ -std=gnu++14 -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Wall -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -D_GLIBCXX_USE_CXX11_ABI=0 -fno-sized-deallocation -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -Os -fno-omit-frame-pointer -funwind-tables -Werror -fPIC -shared -Wl,-z,defs -Wl,--gc-sections -Wl,-h,libxul.so -o libxul.so /builds/worker/workspace/build/src/obj-firefox/toolkit/library/libxul_so.list -lpthread -Wl,-z,noexecstack -Wl,-z,text -Wl,-z,relro -Wl,-z,nocopyreloc -Wl,-Bsymbolic-functions -Wl,--build-id=sha1 /builds/worker/workspace/build/src/toolkit/library/StaticXULComponents.ld -Wl,-rpath-link,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-rpath-link,/usr/local/lib ../../security/nss/lib/crmf/crmf_crmf/libcrmf.a ../../js/src/build/libjs_static.a /builds/worker/workspace/build/src/obj-firefox/x86_64-unknown-linux-gnu/debug/libgkrust.a ../../security/sandbox/linux/libmozsandbox.so ../../config/external/nspr/pr/libnspr4.so ../../config/external/nspr/libc/libplc4.so ../../config/external/nspr/ds/libplds4.so ../../config/external/lgpllibs/liblgpllibs.so ../../security/nss/lib/nss/nss_nss3/libnss3.so ../../security/nss/lib/util/util_nssutil3/libnssutil3.so ../../security/nss/lib/smime/smime_smime3/libsmime3.so ../../config/external/sqlite/libmozsqlite3.so ../../security/nss/lib/ssl/ssl_ssl3/libssl3.so ../../widget/gtk/mozgtk/stub/libmozgtk_stub.so -Wl,--version-script,symverscript -ldl -lrt -lm -lX11 -lX11-xcb -lxcb -lXcomposite -lXcursor -lXdamage -lXext -lXfixes -lXi -lXrender -lpthread -ldl -lc -lfreetype -lfontconfig -Wl,--version-script,/builds/worker/workspace/build/src/build/unix/stdc++compat/hide_std.ld -ldbus-glib-1 -ldbus-1 -lgobject-2.0 -lglib-2.0 -latk-1.0 -lpangocairo-1.0 -lgdk_pixbuf-2.0 -lcairo-gobject -lpango-1.0 -lcairo -lgio-2.0 -lxcb-shm -lpangoft2-1.0 -lXt -lgthread-2.0
[task 2019-02-09T03:58:43.894Z] 03:58:43 INFO - /usr/bin/ld: /builds/worker/workspace/build/src/obj-firefox/x86_64-unknown-linux-gnu/debug/libgkrust.a(mdb.o)(text_env+0x96d): unresolvable AWAVAUATUSHh*HT$( relocation against symbol `stderr@@GLIBC_2.2.5'
[task 2019-02-09T03:58:43.894Z] 03:58:43 INFO - /usr/bin/ld: BFD (GNU Binutils for Debian) 2.22 internal error, aborting at ../../bfd/reloc.c line 443 in bfd_get_reloc_size
[task 2019-02-09T03:58:43.894Z] 03:58:43 INFO - /usr/bin/ld: Please report this bug.
[task 2019-02-09T03:58:43.894Z] 03:58:43 INFO - clang-3.9: error: linker command failed with exit code 1 (use -v to see invocation)
[task 2019-02-09T03:58:43.894Z] 03:58:43 INFO - /builds/worker/workspace/build/src/config/rules.mk:674: recipe for target 'libxul.so' failed
[task 2019-02-09T03:58:43.894Z] 03:58:43 ERROR - make[4]: *** [libxul.so] Error 1
[task 2019-02-09T03:58:43.895Z] 03:58:43 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library'
[task 2019-02-09T03:58:43.895Z] 03:58:43 INFO - /builds/worker/workspace/build/src/config/recurse.mk:74: recipe for target 'toolkit/library/target' failed
[task 2019-02-09T03:58:43.895Z] 03:58:43 ERROR - make[3]: *** [toolkit/library/target] Error 2
[task 2019-02-09T03:58:43.895Z] 03:58:43 INFO - /builds/worker/workspace/build/src/config/recurse.mk:32: recipe for target 'compile' failed
[task 2019-02-09T03:58:43.895Z] 03:58:43 ERROR - make[2]: *** [compile] Error 2
[task 2019-02-09T03:58:43.896Z] 03:58:43 INFO - /builds/worker/workspace/build/src/config/rules.mk:400: recipe for target 'default' failed
[task 2019-02-09T03:58:43.896Z] 03:58:43 ERROR - make[1]: *** [default] Error 2
[task 2019-02-09T03:58:43.896Z] 03:58:43 INFO - client.mk:125: recipe for target 'build' failed
[task 2019-02-09T03:58:43.896Z] 03:58:43 INFO - make: *** [build] Error 2
[task 2019-02-09T03:58:43.924Z] 03:58:43 INFO - 290 compiler warnings present.
[task 2019-02-09T03:58:44.000Z] 03:58:44 INFO - Notification center failed: Install notify-send (usually part of the libnotify package) to get a notification when the build finishes.
[task 2019-02-09T03:58:44.061Z] 03:58:44 ERROR - Return code: 2
[task 2019-02-09T03:58:44.061Z] 03:58:44 WARNING - setting return code to 2
[task 2019-02-09T03:58:44.061Z] 03:58:44 FATAL - 'mach build -v' did not run successfully. Please check log for errors.
[task 2019-02-09T03:58:44.061Z] 03:58:44 FATAL - Running post_fatal callback...
[task 2019-02-09T03:58:44.061Z] 03:58:44 FATAL - Exiting -1
[task 2019-02-09T03:58:44.062Z] 03:58:44 INFO - [mozharness: 2019-02-09 03:58:44.062019Z] Finished build step (failed)

Comment 5

[Mike Hommey [:glandium]]

Funny coincidences.

This job was building fine before the recent rkv changes landed (either bug 1500259 or bug 1490496), but is now busted, but bug 1394825 fixes it.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/cdcbd5b13210
Add a clang-based base toolchains task. r=froydnj

Comment 7

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/cdcbd5b13210

Comment 8

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/1b147d9934f1
Disable base-toolchains-clang jobs for now. a=ccoroiu r=bustage

Comment 9

[Emilio Cobos Álvarez (:emilio)]

I had to disable the jobs. Copying the reason from the commit message from https://hg.mozilla.org/mozilla-central/rev/1b147d9934f1, it's failing in a way that doesn't make any sense to me.

It's hitting the same error as comment 4 in some revisions, but not others. More annoyingly, it hits the error in some revisions while being green in a different revision without any source-code change between them.

This is green:

https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=952b928f1605ad5676ae4ccfd41612b34523bae5

And this is red:

https://treeherder.mozilla.org/#/jobs?repo=autoland&revision=952b928f1605ad5676ae4ccfd41612b34523bae5

And the diff between them is nothing. Which is quite baffling.

I disabled the new jobs on CI for now in order to avoid backing out the stack of dependent patches (bug 1394825, bug 1523071 and bug 1523140) that landed afterwards.

Mike, do you have any idea of how can this happen? Are those builds incremental and are we hitting a strange linker / compiler bug? I wasn't sure if touching CLOBBER would help, but disabling the jobs seemed a less risky approach for now.

Comment 10

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/mozilla-central/rev/50309e384ceb
followup: Make linters happy about a comment. a=ccoroiu r=bustage-fix-bustage

Comment 11

[Mike Hommey [:glandium]]

https://github.com/rust-lang/rust/issues/58393 is what happened. I'll reenable these jobs with sccache disabled.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/8b2e8f177263
Followup: re-enable the tasks without sccache. r=me

Comment 13

[Cristina Coroiu [:ccoroiu]]

https://hg.mozilla.org/mozilla-central/rev/8b2e8f177263