1548408 - ASAN Win64 build fails in WEBRTC

Status: REOPENED

(Firefox Build System :: Toolchains, defect)

Description

[mozillabugs]

Using Mozillabuild 3.2 and the latest ./mach bootstrap (5/1/2019 ~12:00PT) followed by ./mach clobber and ./mach build, and the unmodified source for FF 66.0.3, the build fails with:

[1556748332.448, "build_output", {"line": "fatal error: error in backend: cannot make section .ASAN$GL associative with sectionless symbol ?id@?$collate@D@std@@2V0locale@2@A"}]

[1556748332.473, "build_output", {"line": "mozmake.EXE[4]: *** [c:/mozilla-source/config/rules.mk:1108: Unified_cpp_signaling_src_sdp0.obj] Error 1"}]

My .mozconfig (to which I added ac_add_options --disable-webrtc to work around the bug) is:

ac_add_options --enable-debug-symbols
ac_add_options --disable-optimize
#ac_add_options --disable-crashreporter
mk_add_options MOZ_MAKE_FLAGS="-j3"
ac_add_options --target=x86_64-pc-mingw32
ac_add_options --host=x86_64-pc-mingw32
mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-x64-pc-mingw32

# remove below for non-ASAN builds ------------------------------------------------------------------
# --disable-webrtc is required because otherwise the build dies with:
#
#       [1556748332.448, "build_output", {"line": "fatal error: error in backend: cannot make section .ASAN$GL associative with sectionless symbol ?id@?$collate@D@std@@2V0locale@2@A"}]
#       [1556748332.473, "build_output", {"line": "mozmake.EXE[4]: *** [c:/mozilla-source/config/rules.mk:1108: Unified_cpp_signaling_src_sdp0.obj] Error 1"}]

ac_add_options --enable-address-sanitizer
ac_add_options --disable-jemalloc
ac_add_options --disable-webrtc
export CC="clang-cl.exe"
export CXX="clang-cl.exe"

export LDFLAGS="clang_rt.asan_dynamic-x86_64.lib clang_rt.asan_dynamic_runtime_thunk-x86_64.lib"
CLANG_LIB_DIR="$(cd ~/.mozbuild/clang/lib/clang/*/lib/windows && pwd)"
export MOZ_CLANG_RT_ASAN_LIB_PATH="${CLANG_LIB_DIR}/clang_rt.asan_dynamic-x86_64.dll"
export LIB=$LIB:$CLANG_LIB_DIR
# remove above for non-ASAN builds ------------------------------------------------------------------

#ac_add_options --target=i386-pc-mingw32
#ac_add_options --host=i386-pc-mingw32
#mk_add_options MOZ_OBJDIR=@TOPSRCDIR@/obj-i386-pc-mingw32

Comment 1

[(Away)]

and the unmodified source for FF 66.0.3

Could you be more specific about that -- what's the repo and revision?

Also, could you confirm that grep \'CXX\' obj/obj-x64-pc-mingw32/config.status points to your .mozbuild, and then share the output of ~/.mozbuild/clang/bin/clang-cl --version?

Comment 2

[mozillabugs]

(In reply to David Major [:dmajor] from comment #1)

and the unmodified source for FF 66.0.3

Could you be more specific about that -- what's the repo and revision?

hg summary says

parent: 516429:37ecfd08ffee FIREFOX_66_0_3_BUILD1 FIREFOX_66_0_3_RELEASE
 Bug 1538666: Fix typo in PathCapture::TransformedCopyToBuilder. r=rhunt a=lizzard
branch: default
warning: Watchman unavailable: timed out after waiting 10100ms for read, while executing ('watch', 'c:\\mozilla-source')
commit: 1 modified, 1 unknown (new branch head)
update: 6 new changesets (update)
phases: 516436 draft

Also, could you confirm that grep \'CXX\' obj/obj-x64-pc-mingw32/config.status points to your .mozbuild:

'CXX': 'c:/Users/a/.mozbuild/clang/bin/clang-cl.exe',

which is indeed the clang in my mozbuild tree.

and then share the output of ~/.mozbuild/clang/bin/clang-cl --version?

clang version 8.0.0 (tags/RELEASE_800/rc1 352000)
Target: x86_64-pc-windows-msvc
Thread model: posix
InstalledDir: c:\Users\a\.mozbuild\clang\bin

BTW, after adding ac_add_options --disable-webrtc, the build failed with the same issue in unified_cpp_gfx_gl0.obj, so this bug is a show-stopper.

Comment 3

[(Away)]

clang version 8.0.0 (tags/RELEASE_800/rc1 352000)

This was a pre-release compiler that we stopped using quite some time ago. (I believe you that you ran mach bootstrap very recently, but since you're on the 66 source, bootstrap gave you an old compiler.)

Could you please try either running bootstrap out of a mozilla-central tree, or using this package which I believe is what bootstrap would download: https://queue.taskcluster.net/v1/task/B-YcuzPmTXiZ2KS3jker4A/artifacts/public/build/clang.tar.bz2

Comment 4

[(Away)]

Oh! I think now I remember where I saw this error. I believe a colleague once ran into this issue because they disabled optimization.

For now, don't worry about comment 3. Instead, try removing --disable-optimize, or replacing it with --enable-optimize=-O1.

Comment 5

[mozillabugs]

(In reply to David Major [:dmajor] from comment #4)

Oh! I think now I remember where I saw this error. I believe a colleague once ran into this issue because they disabled optimization.

For now, don't worry about comment 3. Instead, try removing --disable-optimize, or replacing it with --enable-optimize=-O1.

Thank you for the help. I am currently running the comment 3 build. If it fails, I'll try the comment 4 build, though I'm using --disable-optimize so that the debugger produces accurate results. Does -O1 help with that?

Comment 6

[(Away)]

By default our build would use -O2. The -O1 flag does less optimization. I believe the code that hits the build error isn't actually used in the final binary, and -O1 is sufficient to remove that code and avoid the problem, while hopefully not making the debugging experience too terrible.

Comment 7

[mozillabugs]

The new clang itself didn't fix the problem, but using -O1 worked. I'll report how well debugging works tomorrow.

Comment 8

[mozillabugs]

(In reply to mozillabugs from comment #7)

The new clang itself didn't fix the problem, but using -O1 worked. I'll report how well debugging works tomorrow.

The resulting EXE dies during startup. Sigh.

Comment 9

[mozillabugs]

Oops, sorry, I am not sure whether this is fixed.