Closed Bug 1414558 Opened 2 years ago Closed 2 years ago

Build clang from a Docker volume

Categories

(Firefox Build System :: General, defect)

defect
Not set

Tracking

(firefox58 fixed)

RESOLVED FIXED
mozilla58
Tracking Status
firefox58 --- fixed

People

(Reporter: aryx, Assigned: gps)

Details

(Keywords: intermittent-failure)

Attachments

(1 file)

https://treeherder.mozilla.org/logviewer.html#?job_id=142007592&repo=mozilla-inbound

[task 2017-11-03T18:04:53.869Z] [132/139] Linking CXX static library lib/libLLVMTableGen.a
[task 2017-11-03T18:04:53.869Z] [133/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/SubtargetEmitter.cpp.o
[task 2017-11-03T18:04:53.869Z] [134/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86DisassemblerTables.cpp.o
[task 2017-11-03T18:04:53.870Z] [135/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/TableGen.cpp.o
[task 2017-11-03T18:04:53.870Z] [136/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86ModRMFilters.cpp.o
[task 2017-11-03T18:04:53.870Z] [137/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/X86RecognizableInstr.cpp.o
[task 2017-11-03T18:04:53.870Z] [138/139] Building CXX object utils/TableGen/CMakeFiles/obj.llvm-tblgen.dir/CTagsEmitter.cpp.o
[task 2017-11-03T18:04:53.870Z] [139/139] Linking CXX executable bin/llvm-tblgen
[task 2017-11-03T18:04:53.870Z] ninja: build stopped: subcommand failed.
[task 2017-11-03T18:04:53.881Z] Traceback (most recent call last):
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 551, in <module>
[task 2017-11-03T18:04:53.881Z]     build_type, assertions, python_path, gcc_dir, libcxx_include_dir)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 220, in build_one_stage
[task 2017-11-03T18:04:53.881Z]     build_package(build_dir, cmake_args)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 67, in build_package
[task 2017-11-03T18:04:53.881Z]     run_in(package_build_dir, ["ninja", "install"])
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 42, in run_in
[task 2017-11-03T18:04:53.881Z]     check_run(args)
[task 2017-11-03T18:04:53.881Z]   File "./build-clang.py", line 35, in check_run
[task 2017-11-03T18:04:53.881Z]     assert r == 0
[task 2017-11-03T18:04:53.881Z] AssertionError
Earlier in the log, we see:

[task 2017-11-03T18:04:26.152Z] [99/100] Building CXX object tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.o
[task 2017-11-03T18:04:26.152Z] [100/100] Linking CXX executable bin/llvm-config
[task 2017-11-03T18:04:26.152Z] [945/2586] Building CXX object tools/llvm-config/CMakeFiles/llvm-config.dir/llvm-config.cpp.o
[task 2017-11-03T18:04:31.133Z] [946/2586] Linking CXX executable bin/llvm-config
[task 2017-11-03T18:04:51.057Z] FAILED: cd /builds/slave/moz-toolchain/build/stage1/build/NATIVE && /usr/local/bin/cmake --build . --target clang-tblgen --config Release
[task 2017-11-03T18:04:51.057Z] ninja: warning: premature end of file; recovering

[task 2017-11-03T18:04:51.063Z] [113/115] Linking CXX static library lib/libLLVMSupport.a
[task 2017-11-03T18:04:51.063Z] [114/115] Linking CXX static library lib/libLLVMTableGen.a
[task 2017-11-03T18:04:51.064Z] FAILED: : && /usr/local/bin/cmake -E remove lib/libLLVMTableGen.a && /usr/bin/ar qc lib/libLLVMTableGen.a  lib/TableGen/CMakeFiles/LLVMTableGen.dir/Error.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/Main.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/Record.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/SetTheory.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/StringMatcher.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TableGenBackend.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGLexer.cpp.o lib/TableGen/CMakeFiles/LLVMTableGen.dir/TGParser.cpp.o && /usr/bin/ranlib lib/libLLVMTableGen.a && :
[task 2017-11-03T18:04:51.064Z] /usr/bin/ranlib: 'lib/libLLVMTableGen.a': No such file
[task 2017-11-03T18:04:51.064Z] ninja: build stopped: subcommand failed.

This feels like an upstream build system bug.

But, that "premature end of file" warning is concerning. It could possibly be another manifestation of the bad I/O semantics of AUFS. For whatever reason, we're executing things in /builds/slave, which *isn't* a Docker cache or volume. That means AUFS is in play. AUFS is generally bad for performance. So we should definitely move the active building and I/O off AUFS.
Assignee: nobody → gps
Status: NEW → ASSIGNED
Comment on attachment 8925640 [details]
Bug 1414558 - Build Clang toolchain on a Docker volume;

https://reviewboard.mozilla.org/r/196768/#review201994

WFM, thanks for tracking this down.
Attachment #8925640 - Flags: review+
Comment on attachment 8925640 [details]
Bug 1414558 - Build Clang toolchain on a Docker volume;

I would land this automatically, but I see the try push has not completed the toolchain job, so I'll let you land this at your leisure.
Attachment #8925640 - Flags: review?(core-build-config-reviews) → review+
Pushed by gszorc@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/5a9fa46f3cb9
Build Clang toolchain on a Docker volume; r=froydnj
Let's make this bug summary easier to discover.
Summary: Intermittent ninja: build stopped: subcommand failed. → Build clang from a Docker volume
https://hg.mozilla.org/mozilla-central/rev/5a9fa46f3cb9
Status: ASSIGNED → RESOLVED
Closed: 2 years ago
Resolution: --- → FIXED
Target Milestone: --- → mozilla58
Duplicate of this bug: 1423775
Product: Core → Firefox Build System
You need to log in before you can comment on or make changes to this bug.