1839299 - Use a finer-grained strategy than stack walking suppressions to avoid stack walking deadlock on Windows x64

Status: RESOLVED FIXED

(Core :: Gecko Profiler, enhancement, P2)

Description

[Yannis Juglaret [:yannis]]

Stack walking on Windows x64 and aarch64 relies on RtlLookupFunctionEntry, which acquires up to two ntdll-internal SRW locks as shared, namely (according to PDB symbols):

• LdrpInvertedFunctionTableSRWLock: always, to match with a function from an executable file;
• RtlpDynamicFunctionTableLock: if that fails, to match with function tables for dynamic code.

We currently avoid deadlock situations by using stack walk suppressions, which means that we guard some paths, such that if any thread is within such path at stack walk time we do not attempt to walk the stack. It is a coarse-grained strategy that prevents us from profiling some situations (e.g. DLL loading) as best as we could theoretically. We must guard all paths that could end up acquiring the locks mentioned above, and we are likely to miss some of those paths, which has led and can still lead to deadlock when using the Firefox Profiler, or when using Nightly through Background Hang Reporter.

We could try a fine-grained strategy instead, by collecting pointers to the ntdll-internal SRW locks mentioned above, which would let us use AcquireSRWLockShared and TryAcquireSRWLockShared on these locks, which should solve most of our problems.

See also the discussion starting with bug 1836225 comment 2.

Comment 1

[Yannis Juglaret [:yannis]]

Attachment: Bug 1839299 - Add support for rex lea and lock rex cmpxchg. r=handyman

We have hooking code for the LEA instruction in x86_64, but not when it
is REX-prefixed. We need the REX-prefixed variant to hook
RtlReleaseSRWLockShared from ntdll.

Depends on D181628

Comment 2

[Yannis Juglaret [:yannis]]

Attachment: Bug 1839299 - Use a finer-grained strategy to protect RtlLookupFunctionEntry against deadlocks. r=handyman

On 64-bit Windows (x86_64, aarch64), stack walking relies on
RtlLookupFunctionEntry to navigate from one frame to the next. This
function acquires up to two ntdll internal locks when it is called.

The profiler and the background hang monitor both need to walk the
stacks of suspended threads. This can lead to deadlock situations,
which so far we have avoided with stack walk suppressions. We guard some
critical paths to mark them as suppressing stack walk, and we forbid
stack walking when any thread is currently on such path.

While stack walk suppression has helped remove most deadlock situations,
some can remain because it is hard to detect and manually annotate all
the paths that could lead to a deadlock situation. Another drawback is
that stack walk suppression disables stack walking for much larger
portions of code than required. For example, we disable stack walking
for LdrLoadDll, so we cannot collect stacks while we are loading a DLL.
Yet, the lock that could lead to a deadlock situation is only held
during a very small portion of the whole time spent in LdrLoadDll.

This patch addresses these two issues by implementing a finer-grained
strategy to avoid deadlock situations. We hook ntdll lock acquisition
and release functions in order to collect pointers to the two
problematic locks. This allows us to play with these locks, such that
we can guarantee safe stack walking with no deadlock.

If we fail to collect pointers to the locks, we fall back to using stack
walk suppressions like before. This way we get the best of both worlds:
if we are confident that the situation is under control, we will use the
new strategy and get better profiler accuracy and no deadlock; in case
of doubt, we can still use the profiler thanks to stack walk
suppressions.

Depends on D182553

Comment 3

[Yannis Juglaret [:yannis]]

Bug 1823412 was a good example of a bug where capturing the stack during DLL loads could have been helpful. Back then I shared some profiles in bug 1823412 comment 9. Here is what startup profiles would look like for this same bug if taken using the proposed strategy:

• before the patch for bug 1823412: https://share.firefox.dev/3NpwLUU
• after the patch for bug 1823412: https://share.firefox.dev/3psjWRQ

Comment 4

[Yannis Juglaret [:yannis]]

Attachment: Bug 1839299 - Use a finer-grained strategy to protect RtlLookupFunctionEntry against deadlocks. r=rkraesig,#win-reviewers

On 64-bit Windows (x86_64, aarch64), stack walking relies on
RtlLookupFunctionEntry to navigate from one frame to the next. This
function acquires up to two ntdll internal locks when it is called.

The profiler and the background hang monitor both need to walk the
stacks of suspended threads. This can lead to deadlock situations,
which so far we have avoided with stack walk suppressions. We guard some
critical paths to mark them as suppressing stack walk, and we forbid
stack walking when any thread is currently on such path.

While stack walk suppression has helped remove most deadlock situations,
some can remain because it is hard to detect and manually annotate all
the paths that could lead to a deadlock situation. Another drawback is
that stack walk suppression disables stack walking for much larger
portions of code than required. For example, we disable stack walking
for LdrLoadDll, so we cannot collect stacks while we are loading a DLL.
Yet, the lock that could lead to a deadlock situation is only held
during a very small portion of the whole time spent in LdrLoadDll.

This patch addresses these two issues by implementing a finer-grained
strategy to avoid deadlock situations. We acquire the pointers to the
internel ntdll locks through a single-stepped execution of
RtlLookupFunctionEntry. This allows us to try to acquire the locks
non-blockingly so that we can guarantee safe stack walking with no
deadlock.

If we fail to collect pointers to the locks, we fall back to using stack
walk suppressions like before. This way we get the best of both worlds:
if we are confident that the situation is under control, we will use the
new strategy and get better profiler accuracy and no deadlock; in case
of doubt, we can still use the profiler thanks to stack walk
suppressions.

Comment 5

[Yannis Juglaret [:yannis]]

Moving aarch64 support to bug 1921409.

Comment 6

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/e6475b7e0602
Use a finer-grained strategy to protect RtlLookupFunctionEntry against deadlocks. r=win-reviewers,rkraesig

Comment 7

[agoloman]

Backed out for causing instrumented build bustages.

• Backout link
• Push with failures
• Failure Log
• Failure line:

[task 2024-10-17T10:44:41.885Z] 10:44:41     INFO -  TEST-PASS | autowinchecksec.py | ../../../../dist/bin/updater.exe succeeded
[task 2024-10-17T10:44:41.885Z] 10:44:41     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/mozapps/update/updater'
[task 2024-10-17T10:44:41.931Z] 10:44:41     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/dom/media/fake-cdm'
[task 2024-10-17T10:44:41.932Z] 10:44:41     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/dom/media/fake-cdm'
[task 2024-10-17T10:44:42.095Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.096Z] 10:44:42     INFO -  /builds/worker/fetches/nasm/nasm -o vp9lpf.obj -f win64 -F cv8 -D__x86_64__ -DPIC -DWIN64 -DMSVC -Pconfig_win64.asm -I/builds/worker/checkouts/gecko/media/ffvpx/ -I/builds/worker/checkouts/gecko/media/ffvpx/libavcodec/x86/ -I/builds/worker/checkouts/gecko/media/ffvpx/libavutil/x86/   ../../../../../../../../builds/worker/checkouts/gecko/media/ffvpx/libavcodec/x86/vp9lpf.asm
[task 2024-10-17T10:44:42.097Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.097Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.097Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.097Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.098Z] 10:44:42     INFO -  media/ffvpx/libavcodec/x86/vp9mc.obj
[task 2024-10-17T10:44:42.098Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/media/ffvpx/libavcodec/x86'
[task 2024-10-17T10:44:42.225Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.225Z] 10:44:42     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/config/create_rc.py 'private_browsing.exe' '/builds/worker/checkouts/gecko/browser/app/pbproxy/pbproxy.rc'
[task 2024-10-17T10:44:42.225Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.226Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.226Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.227Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.227Z] 10:44:42     INFO -  browser/app/pbproxy/private_browsing.exe.res
[task 2024-10-17T10:44:42.228Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.277Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.277Z] 10:44:42     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/config/create_rc.py 'mozglue.dll' ''
[task 2024-10-17T10:44:42.278Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.291Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.292Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.312Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.312Z] 10:44:42     INFO -  mozglue/build/mozglue.dll.res
[task 2024-10-17T10:44:42.312Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.423Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/widget/windows'
[task 2024-10-17T10:44:42.427Z] 10:44:42     INFO -  /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -fms-compatibility-version=19.39 -std:c++17 -Xclang -ivfsoverlay -Xclang /builds/worker/fetches/vs/overlay.yaml -FoTSFTextStore.obj -c  -I/builds/worker/workspace/obj-build/dist/stl_wrappers -guard:cf -DNDEBUG=1 -DTRIMMED=1 -DUNICODE -D_UNICODE -D_CRT_RAND_S -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -D_SECURE_ATL -DCHROMIUM_BUILD -DU_STATIC_IMPLEMENTATION -DWIN32 -D_WIN32 -D_WINDOWS -DWIN32_LEAN_AND_MEAN -DMOZ_UNICODE '-DMOZ_APP_NAME="firefox"' '-DMOZ_TOAST_APP_NAME="Firefox"' -DWINAPI_NO_BUNDLED_LIBRARIES -DMOZ_HAS_MOZGLUE -DMOZILLA_INTERNAL_API -DIMPL_LIBXUL -DMOZ_SUPPORT_LEAKCHECKING -DSTATIC_EXPORTABLE_JS_API -I/builds/worker/checkouts/gecko/widget/windows -I/builds/worker/workspace/obj-build/widget/windows -I/builds/worker/workspace/obj-build/ipc/ipdl/_ipdlheaders -I/builds/worker/checkouts/gecko/ipc/chromium/src -I/builds/worker/checkouts/gecko/gfx/cairo/cairo/src -I/builds/worker/checkouts/gecko/layout/forms -I/builds/worker/checkouts/gecko/layout/generic -I/builds/worker/checkouts/gecko/layout/style -I/builds/worker/checkouts/gecko/layout/xul -I/builds/worker/checkouts/gecko/toolkit/components/jsoncpp/include -I/builds/worker/checkouts/gecko/toolkit/xre -I/builds/worker/checkouts/gecko/widget -I/builds/worker/checkouts/gecko/widget/headless -I/builds/worker/checkouts/gecko/xpcom/base -I/builds/worker/workspace/obj-build/dist/include -I/builds/worker/workspace/obj-build/dist/include/nspr -I/builds/worker/workspace/obj-build/dist/include/nss -MD -DMOZILLA_CLIENT -FI /builds/worker/workspace/obj-build/mozilla-config.h -Zc:sizedDealloc- -Gy -Zc:inline -Gw -D_SILENCE_TR1_NAMESPACE_DEPRECATION_WARNING -TP -GR- -D_HAS_EXCEPTIONS=0 -fcrash-diagnostics-dir=/builds/worker/artifacts -Z7 -Xclang -load -Xclang /builds/worker/workspace/obj-build/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O2 -Oy -Werror -W3 -Wbitfield-enum-conversion -Wempty-body -Wformat-type-confusion -Wignored-qualifiers -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtautological-constant-in-range-compare -Wtype-limits -Wno-error=tautological-type-limit-compare -Wunreachable-code -Wunreachable-code-return -Wunused-but-set-parameter -Wno-invalid-offsetof -Wclass-varargs -Wempty-init-stmt -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wno-range-loop-analysis -Wenum-compare-conditional -Wenum-float-conversion -Wvolatile -Wno-deprecated-anon-enum-enum-conversion -Wno-deprecated-enum-enum-conversion -Wno-deprecated-this-capture -Wcomma -Wimplicit-fallthrough -Wstring-conversion -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=free-nonheap-object -Wno-error=atomic-alignment -Wno-error=deprecated-builtins -Wno-unknown-pragmas -Wno-ignored-pragmas -Wno-deprecated-declarations -Wno-microsoft-enum-value -Wno-microsoft-include -Wno-invalid-noreturn -Wno-inconsistent-missing-override -Wno-implicit-exception-spec-mismatch -Wno-microsoft-exception-spec -Wno-unused-local-typedef -Wno-ignored-attributes -Wno-used-but-marked-unused -Wno-psabi -Wthread-safety -Wno-error=builtin-macro-redefined -Wno-vla-cxx-extension -Wno-unknown-warning-option -fno-strict-aliasing -Xclang -ffp-contract=off -DNS_FREE_PERMANENT_DATA=1 /clang:-fprofile-generate -mllvm -pgo-temporal-instrumentation -mllvm -enable-name-compression=false -Xclang -MP -Xclang -dependency-file -Xclang .deps/TSFTextStore.obj.pp -Xclang -MT -Xclang TSFTextStore.obj   /builds/worker/checkouts/gecko/widget/windows/TSFTextStore.cpp
[task 2024-10-17T10:44:42.428Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/widget/windows'
[task 2024-10-17T10:44:42.435Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/widget/windows'
[task 2024-10-17T10:44:42.436Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/widget/windows'
[task 2024-10-17T10:44:42.467Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/dom/media/gmp-plugin-openh264'
[task 2024-10-17T10:44:42.468Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/dom/media/gmp-plugin-openh264'
[task 2024-10-17T10:44:42.815Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.816Z] 10:44:42     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/config/create_res.py -DNDEBUG=1 -DTRIMMED=1 -DUNICODE=1 '-DMOZ_APP_NAME=L"firefox"' '-DPBMODE_ICO="/builds/worker/checkouts/gecko/browser/branding/nightly/pbmode.ico"' -I/builds/worker/checkouts/gecko/browser/app/pbproxy -I/builds/worker/workspace/obj-build/browser/app/pbproxy -I/builds/worker/checkouts/gecko/toolkit/xre -I/builds/worker/workspace/obj-build/dist/include  -o private_browsing.exe.res private_browsing.exe.rc
[task 2024-10-17T10:44:42.817Z] 10:44:42     INFO -  Executing: /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -E -fms-compatibility-version=19.39 -std:c++17 -Xclang -ivfsoverlay -Xclang /builds/worker/fetches/vs/overlay.yaml -D NDEBUG=1 -D TRIMMED=1 -D UNICODE=1 -D MOZ_APP_NAME=L"firefox" -D PBMODE_ICO="/builds/worker/checkouts/gecko/browser/branding/nightly/pbmode.ico" -I /builds/worker/checkouts/gecko/browser/app/pbproxy -I /builds/worker/workspace/obj-build/browser/app/pbproxy -I /builds/worker/checkouts/gecko/toolkit/xre -I /builds/worker/workspace/obj-build/dist/include -DRC_INVOKED private_browsing.exe.rc -D _AMD64_
[task 2024-10-17T10:44:42.818Z] 10:44:42     INFO -  Executing: /builds/worker/fetches/clang/bin/llvm-rc -D NDEBUG=1 -D TRIMMED=1 -D UNICODE=1 -D MOZ_APP_NAME=L"firefox" -D PBMODE_ICO="/builds/worker/checkouts/gecko/browser/branding/nightly/pbmode.ico" -I /builds/worker/checkouts/gecko/browser/app/pbproxy -I /builds/worker/workspace/obj-build/browser/app/pbproxy -I /builds/worker/checkouts/gecko/toolkit/xre -I /builds/worker/workspace/obj-build/dist/include -c 1252 -Foprivate_browsing.exe.res /tmp/tmp_zyap_qk.rc
[task 2024-10-17T10:44:42.818Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.823Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.823Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.836Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.836Z] 10:44:42     INFO -  browser/app/pbproxy/private_browsing.exe
[task 2024-10-17T10:44:42.836Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.875Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.876Z] 10:44:42     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/config/create_res.py -DNDEBUG=1 -DTRIMMED=1 -DIMPL_MFBT -DMOZ_HAS_MOZGLUE -I/builds/worker/checkouts/gecko/mozglue/build -I/builds/worker/workspace/obj-build/mozglue/build -I/builds/worker/checkouts/gecko/memory/build -I/builds/worker/workspace/obj-build/dist/include  -o mozglue.dll.res mozglue.dll.rc
[task 2024-10-17T10:44:42.876Z] 10:44:42     INFO -  Executing: /builds/worker/fetches/sccache/sccache /builds/worker/fetches/clang/bin/clang-cl -E -fms-compatibility-version=19.39 -std:c++17 -Xclang -ivfsoverlay -Xclang /builds/worker/fetches/vs/overlay.yaml -D NDEBUG=1 -D TRIMMED=1 -D IMPL_MFBT -D MOZ_HAS_MOZGLUE -I /builds/worker/checkouts/gecko/mozglue/build -I /builds/worker/workspace/obj-build/mozglue/build -I /builds/worker/checkouts/gecko/memory/build -I /builds/worker/workspace/obj-build/dist/include -DRC_INVOKED mozglue.dll.rc -D _AMD64_
[task 2024-10-17T10:44:42.877Z] 10:44:42     INFO -  Executing: /builds/worker/fetches/clang/bin/llvm-rc -D NDEBUG=1 -D TRIMMED=1 -D IMPL_MFBT -D MOZ_HAS_MOZGLUE -I /builds/worker/checkouts/gecko/mozglue/build -I /builds/worker/workspace/obj-build/mozglue/build -I /builds/worker/checkouts/gecko/memory/build -I /builds/worker/workspace/obj-build/dist/include -c 1252 -Fomozglue.dll.res /tmp/tmpg5jb2dki.rc
[task 2024-10-17T10:44:42.877Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.880Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.881Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.883Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.883Z] 10:44:42     INFO -  mozglue/build/mozglue.dll
[task 2024-10-17T10:44:42.883Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.892Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.892Z] 10:44:42     INFO -  rm -f ../../dist/bin/mozglue.dll
[task 2024-10-17T10:44:42.894Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.950Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.953Z] 10:44:42     INFO -  /builds/worker/fetches/clang/bin/lld-link -NOLOGO -DLL -OUT:../../dist/bin/mozglue.dll -PDB:mozglue.pdb -SUBSYSTEM:WINDOWS,10.0 -MACHINE:X64 -IMPLIB:mozglue.lib @/builds/worker/workspace/obj-build/mozglue/build/mozglue_dll.list mozglue.dll.res   -DEBUG -PDBALTPATH:%_PDB% -OPT:REF,ICF -guard:cf,nolongjmp -DEF:../../../../checkouts/gecko/mozglue/build/mozglue.def -DELAYLOAD:advapi32.dll -DELAYLOAD:dbghelp.dll -DELAYLOAD:oleaut32.dll -DELAYLOAD:ole32.dll -DELAYLOAD:user32.dll -DELAYLOAD:version.dll -DELAYLOAD:winmm.dll  ../../build/pure_virtual/pure_virtual.lib    dbghelp.lib oleaut32.lib ole32.lib mincore.lib crypt32.lib ntdll.lib version.lib wintrust.lib delayimp.lib advapi32.lib user32.lib winmm.lib uuid.lib
[task 2024-10-17T10:44:42.955Z] 10:44:42    ERROR -  lld-link: error: undefined symbol: public: __cdecl mozilla::AutoOnSingleStepCallback::AutoOnSingleStepCallback(class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *)
[task 2024-10-17T10:44:42.956Z] 10:44:42     INFO -  >>> referenced by /builds/worker/workspace/obj-build/dist/include/mozilla/WindowsDiagnostics.h:122
[task 2024-10-17T10:44:42.959Z] 10:44:42     INFO -  >>>               ../../toolkit/xre/dllservices/mozglue/Unified_cpp_dllservices_mozglue0.obj:(enum mozilla::WindowsDiagnosticsError __cdecl mozilla::CollectSingleStepData<void (__cdecl *)(void)>(void (__cdecl *)(void), class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *))
[task 2024-10-17T10:44:42.959Z] 10:44:42    ERROR -  lld-link: error: undefined symbol: long __cdecl mozilla::SingleStepExceptionHandler(struct _EXCEPTION_POINTERS *)
[task 2024-10-17T10:44:42.960Z] 10:44:42     INFO -  >>> referenced by /builds/worker/workspace/obj-build/dist/include/mozilla/WindowsDiagnostics.h:125
[task 2024-10-17T10:44:42.960Z] 10:44:42     INFO -  >>>               ../../toolkit/xre/dllservices/mozglue/Unified_cpp_dllservices_mozglue0.obj:(enum mozilla::WindowsDiagnosticsError __cdecl mozilla::CollectSingleStepData<void (__cdecl *)(void)>(void (__cdecl *)(void), class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *))
[task 2024-10-17T10:44:42.961Z] 10:44:42    ERROR -  lld-link: error: undefined symbol: void __cdecl mozilla::EnableTrapFlag(void)
[task 2024-10-17T10:44:42.961Z] 10:44:42     INFO -  >>> referenced by /builds/worker/workspace/obj-build/dist/include/mozilla/WindowsDiagnostics.h:130
[task 2024-10-17T10:44:42.962Z] 10:44:42     INFO -  >>>               ../../toolkit/xre/dllservices/mozglue/Unified_cpp_dllservices_mozglue0.obj:(enum mozilla::WindowsDiagnosticsError __cdecl mozilla::CollectSingleStepData<void (__cdecl *)(void)>(void (__cdecl *)(void), class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *))
[task 2024-10-17T10:44:42.963Z] 10:44:42    ERROR -  lld-link: error: undefined symbol: void __cdecl mozilla::DisableTrapFlag(void)
[task 2024-10-17T10:44:42.964Z] 10:44:42     INFO -  >>> referenced by /builds/worker/workspace/obj-build/dist/include/mozilla/WindowsDiagnostics.h:132
[task 2024-10-17T10:44:42.965Z] 10:44:42     INFO -  >>>               ../../toolkit/xre/dllservices/mozglue/Unified_cpp_dllservices_mozglue0.obj:(enum mozilla::WindowsDiagnosticsError __cdecl mozilla::CollectSingleStepData<void (__cdecl *)(void)>(void (__cdecl *)(void), class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *))
[task 2024-10-17T10:44:42.965Z] 10:44:42    ERROR -  lld-link: error: undefined symbol: public: __cdecl mozilla::AutoOnSingleStepCallback::~AutoOnSingleStepCallback(void)
[task 2024-10-17T10:44:42.966Z] 10:44:42     INFO -  >>> referenced by /builds/worker/workspace/obj-build/dist/include/mozilla/WindowsDiagnostics.h:136
[task 2024-10-17T10:44:42.968Z] 10:44:42     INFO -  >>>               ../../toolkit/xre/dllservices/mozglue/Unified_cpp_dllservices_mozglue0.obj:(enum mozilla::WindowsDiagnosticsError __cdecl mozilla::CollectSingleStepData<void (__cdecl *)(void)>(void (__cdecl *)(void), class std::function<bool __cdecl(void *, struct _CONTEXT *)>, void *))
[task 2024-10-17T10:44:42.969Z] 10:44:42    ERROR -  gmake[4]: *** [/builds/worker/checkouts/gecko/config/rules.mk:539: ../../dist/bin/mozglue.dll] Error 1
[task 2024-10-17T10:44:42.969Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/mozglue/build'
[task 2024-10-17T10:44:42.970Z] 10:44:42     INFO -  gmake[4]: Target 'target' not remade because of errors.
[task 2024-10-17T10:44:42.970Z] 10:44:42    ERROR -  gmake[3]: *** [/builds/worker/checkouts/gecko/config/recurse.mk:72: mozglue/build/target] Error 2
[task 2024-10-17T10:44:42.975Z] 10:44:42     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:42.976Z] 10:44:42     INFO -  /builds/worker/fetches/clang/bin/lld-link -OUT:../../../dist/bin/private_browsing.exe -PDB:private_browsing.pdb -IMPLIB:private_browsing.lib -SUBSYSTEM:WINDOWS,10.0 -STACK:8388608  -DEBUG -PDBALTPATH:%_PDB% -OPT:REF,ICF -guard:cf,nolongjmp   pbproxy.obj private_browsing.exe.res ../../../build/pure_virtual/pure_virtual.lib  shlwapi.lib user32.lib
[task 2024-10-17T10:44:42.976Z] 10:44:42     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
< >
[task 2024-10-17T10:44:44.984Z] 10:44:44     INFO -  /builds/worker/fetches/clang/bin/lld-link -OUT:../../../dist/bin/maintenanceservice.exe -PDB:maintenanceservice.pdb -IMPLIB:maintenanceservice.lib -ENTRY:wmainCRTStartup -SUBSYSTEM:WINDOWS,10.0 -STACK:8388608  -DEBUG -PDBALTPATH:%_PDB% -OPT:REF,ICF -guard:cf,nolongjmp   maintenanceservice.obj servicebase.obj serviceinstall.obj workmonitor.obj maintenanceservice.exe.res ../../../toolkit/mozapps/update/common/updatecommon.lib ../../../build/pure_virtual/pure_virtual.lib  advapi32.lib ole32.lib rpcrt4.lib shell32.lib crypt32.lib wintrust.lib comctl32.lib ws2_32.lib shlwapi.lib user32.lib userenv.lib uuid.lib
[task 2024-10-17T10:44:44.984Z] 10:44:44     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/components/maintenanceservice'
[task 2024-10-17T10:44:44.984Z] 10:44:44     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/toolkit/components/maintenanceservice'
[task 2024-10-17T10:44:44.984Z] 10:44:44     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/components/maintenanceservice'
[task 2024-10-17T10:44:44.989Z] 10:44:44     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/toolkit/components/maintenanceservice'
[task 2024-10-17T10:44:44.989Z] 10:44:44     INFO -  gmake[4]: Leaving directory '/builds/worker/workspace/obj-build/toolkit/components/maintenanceservice'
[task 2024-10-17T10:44:45.124Z] 10:44:45     INFO -  gmake[4]: Entering directory '/builds/worker/workspace/obj-build/browser/app/pbproxy'
[task 2024-10-17T10:44:45.124Z] 10:44:45     INFO -  /builds/worker/.mozbuild/srcdirs/gecko-8a5b87fe5d69/_virtualenvs/build/bin/python /builds/worker/checkouts/gecko/build/win32/autowinchecksec.py ../../../dist/bin/private_browsing.exe
[task 2024-10-17T10:44:45.125Z] 10:44:45     INFO -  Warn: large load config, probably contains undocumented fields
[task 2024-10-17T10:44:45.125Z] 10:44:45     INFO -  TEST-PASS | autowinchecksec.py | ../../../dist/bin/private_browsing.exe succeeded

Comment 8

[Yannis Juglaret [:yannis]]

Sorry - my mistake. Will reland if this target works.

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/192b5e223104
Use a finer-grained strategy to protect RtlLookupFunctionEntry against deadlocks. r=win-reviewers,rkraesig

Comment 10

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/192b5e223104