1414558 - Build clang from a Docker volume

Status: RESOLVED FIXED

(Firefox Build System :: General, defect)

Description

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://treeherder.mozilla.org/logviewer.html#?job_id=142007592&repo=mozilla-inbound

[task 2017-11-03T18:04:53.869Z] [132/139] Linking CXX static library lib/libLLVMTableGen.a
[task 2017-11-03T18:04:53.869Z] [133/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SubtargetEmitter.cpp.o
[task 2017-11-03T18:04:53.869Z] [134/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86DisassemblerTables.cpp.o
[task 2017-11-03T18:04:53.870Z] [135/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/TableGen.cpp.o
[task 2017-11-03T18:04:53.870Z] [136/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86ModRMFilters.cpp.o
[task 2017-11-03T18:04:53.870Z] [137/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86RecognizableInstr.cpp.o
[task 2017-11-03T18:04:53.870Z] [138/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CTagsEmitter.cpp.o
[task 2017-11-03T18:04:53.870Z] [139/139] Linking CXX executable bin/llvm-tblgen
[task 2017-11-03T18:04:53.870Z] ninja: build stopped: subcommand failed.
[task 2017-11-03T18:04:53.881Z] Traceback (most recent call last):
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 551, in <module>
[task 2017-11-03T18:04:53.881Z]     build_type, assertions, python_path, gcc_dir, libcxx_include_dir)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 220, in build_one_stage
[task 2017-11-03T18:04:53.881Z]     build_package(build_dir, cmake_args)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 67, in build_package
[task 2017-11-03T18:04:53.881Z]     run_in(package_build_dir, ["ninja", "install"])
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 42, in run_in
[task 2017-11-03T18:04:53.881Z]     check_run(args)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 35, in check_run
[task 2017-11-03T18:04:53.881Z]     assert r == 0
[task 2017-11-03T18:04:53.881Z] AssertionError

Comment 1

[Intermittent Failures Robot]

1 failures in 857 pushes (0.001 failures/push) were associated with this bug in the last 7 days.    

Repository breakdown:
* mozilla-inbound: 1

Platform breakdown:
* toolchains: 1

For more details, see:
https://brasstacks.mozilla.com/orangefactor/?display=Bug&bugid=1414558&startday=2017-10-30&endday=2017-11-05&tree=all

Comment 2

[Gregory Szorc [:gps]]

Earlier in the log, we see:

[task 2017-11-03T18:04:26.152Z] [99/100] Building CXX object tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.o
[task 2017-11-03T18:04:26.152Z] [100/100] Linking CXX executable bin/llvm-config
[task 2017-11-03T18:04:26.152Z] [945/2586] Building CXX object tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.o
[task 2017-11-03T18:04:31.133Z] [946/2586] Linking CXX executable bin/llvm-config
[task 2017-11-03T18:04:51.057Z] FAILED: cd /builds/slave/moz-toolchain/build/stage1/build/NATIVE && /usr/local/bin/cmake --build . --target clang-tblgen --config Release
[task 2017-11-03T18:04:51.057Z] ninja: warning: premature end of file; recovering

[task 2017-11-03T18:04:51.063Z] [113/115] Linking CXX static library lib/libLLVMSupport.a
[task 2017-11-03T18:04:51.063Z] [114/115] Linking CXX static library lib/libLLVMTableGen.a
[task 2017-11-03T18:04:51.064Z] FAILED: : && /usr/local/bin/cmake -E remove lib/libLLVMTableGen.a && /usr/bin/ar qc lib/libLLVMTableGen.a  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Error.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/Main.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/Record.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/SetTheory.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/StringMatcher.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TableGenBackend.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGLexer.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGParser.cpp.o && /usr/bin/ranlib lib/libLLVMTableGen.a && :
[task 2017-11-03T18:04:51.064Z] /usr/bin/ranlib: 'lib/libLLVMTableGen.a': No such file
[task 2017-11-03T18:04:51.064Z] ninja: build stopped: subcommand failed.

This feels like an upstream build system bug.

But, that "premature end of file" warning is concerning. It could possibly be another manifestation of the bad I/O semantics of AUFS. For whatever reason, we're executing things in /builds/slave, which *isn't* a Docker cache or volume. That means AUFS is in play. AUFS is generally bad for performance. So we should definitely move the active building and I/O off AUFS.

Comment 3

[Gregory Szorc [:gps]]

Attachment: Bug 1414558 - Build Clang toolchain on a Docker volume;

Before, the build root was not in a Docker cache or volume. With
current Docker works, that meant AUFS. We know AUFS is slow under
I/O load and can cause random failures due to missing data after
writes.

This commit changes the build root to a known Docker volume, which
will be backed by EXT4 and won't have the problems of AUFS.

Review commit: https://reviewboard.mozilla.org/r/196768/diff/#index_header
See other reviews: https://reviewboard.mozilla.org/r/196768/

Comment 4

[Nathan Froyd [:froydnj]]

Comment on attachment 8925640 [details]
Bug 1414558 - Build Clang toolchain on a Docker volume;

https://reviewboard.mozilla.org/r/196768/#review201994

WFM, thanks for tracking this down.

Comment 5

[Nathan Froyd [:froydnj]]

Comment on attachment 8925640 [details]
Bug 1414558 - Build Clang toolchain on a Docker volume;

I would land this automatically, but I see the try push has not completed the toolchain job, so I'll let you land this at your leisure.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/5a9fa46f3cb9
Build Clang toolchain on a Docker volume; r=froydnj

Comment 7

[Gregory Szorc [:gps]]

Let's make this bug summary easier to discover.

Comment 8

[Andreea Pavel [:apavel]]

https://hg.mozilla.org/mozilla-central/rev/5a9fa46f3cb9