1912078 - config5 indexer is hanging every day on wubkat build

Status: RESOLVED FIXED

(Webtools :: Searchfox, defect)

Description

[Andrew Sutherland [:asuth] (he/him)]

Our last good config5 release5 run apparently was July 25th; since then config5 has been hanging in what appears to be a case of resource-exhaustion to an extent that our self-terminating cron job logic is unable to fire and generate an error. After being unable to login to today's run either, I stopped it (which took a while, but it did stop on its own) and retrieved the following notable tail from the index-log:

[ 14%] Built target WTF_CopyHeaders
[ 14%] Generating ../../bmalloc/Headers/bmalloc/pas_virtual_range.h
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/HeapConstants.cpp.o
[ 14%] Generating ../../bmalloc/Headers/bmalloc/pas_virtual_range_min_heap.h
[ 14%] Generating ../../bmalloc/Headers/bmalloc/pas_zero_mode.h
[ 14%] Generating ../../bmalloc/Headers/bmalloc/thingy_heap_config.h
[ 14%] Generating ../../bmalloc/Headers/bmalloc/thingy_heap.h
[ 14%] Generating ../../bmalloc/Headers/bmalloc/thingy_heap_prefix.h
[ 14%] Built target bmalloc_CopyHeaders
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/HeapKind.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/android_util.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/angleutils.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/IsoHeap.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/IsoHeapImpl.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/base/anglebase/sha1.cc.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/IsoMallocFallback.cpp.o
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/debug.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/entry_points_enum_autogen.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/event_tracer.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/mathutil.cpp.o
[ 14%] Built target WebCoreBindings
[ 14%] Generate bindings (WebCoreTestSupportBindings)
Killed
make[2]: *** [Source/bmalloc/CMakeFiles/bmalloc.dir/build.make:76: Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/AllIsoHeaps.cpp.o] Error 137
make[2]: *** Waiting for unfinished jobs....
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/matrix_utils.cpp.o
[ 14%] Built target WebCoreTestSupportBindings
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/common/platform_helpers.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
Killed
make[2]: *** [Source/bmalloc/CMakeFiles/bmalloc.dir/build.make:202: Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/Gigacage.cpp.o] Error 137

Given that a kill was observed, this leads me to believe it's a resource exhaustion problem and I have made minor changes to allow us to run a larger instance for release5 (m5d.4xlarge versus the existing m5d.2xlarge) in order to provide more headroom if that's the problem. Also, because we run the full webkit build on the indexer itself, it arguably does make sense to give it a larger instance type than if we were only running the searchfox logic. (searchfox can do a lot of things in parallel, but not everything in parallel, whereas for a large C++ codebase build, we proportionately will spend a much larger time being able to saturate the system.) In particular, we expect the marginal cost increase to be minimal, or potentially even negative if the machine had previously been experiencing swap delays.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Attachment: htop on hang

Still happened; seeing some very high per-core memory usages, and one of which did seem to get OOM-killed:

[ 14%] Built target WTF_CopyHeaders
[ 14%] Generating ../../bmalloc/Headers/bmalloc/thingy_heap_prefix.h
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/Macro.cpp.o
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/MacroExpander.cpp.o
[ 14%] Built target bmalloc_CopyHeaders
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/Preprocessor.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/Token.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/preprocessor_lex_autogen.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/preprocessor/preprocessor_tab_autogen.cpp.o
[ 14%] Built target WebCoreBindings
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/LargeMap.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/OutputGLSLBase.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/OutputESSL.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/TranslatorESSL.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/ImmutableString_ESSL_autogen.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/SymbolTable_ESSL_autogen.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/Logging.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/Mutex.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/OutputGLSL.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/ObjectType.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/ObjectTypeTable.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/BuiltInFunctionEmulatorGLSL.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
Killed
make[2]: *** [Source/bmalloc/CMakeFiles/bmalloc.dir/build.make:76: Source/bmalloc/CMakeFiles/bmalloc.dir/bmalloc/AllIsoHeaps.cpp.o] Error 137
make[2]: *** Waiting for unfinished jobs....
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/tree_ops/glsl/RegenerateStructNames.cpp.o
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/ExtensionGLSL.cpp.o
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/TranslatorGLSL.cpp.o
[ 14%] Building CXX object Source/ThirdParty/ANGLE/CMakeFiles/ANGLE.dir/src/compiler/translator/glsl/VersionGLSL.cpp.o
[ 14%] Generate bindings (WebCoreTestSupportBindings)

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: htop on hang with ENABLE_WEBGL=OFF

I tried disabling ANGLE figuring maybe there was just something pathological going on with that subsystem, but either the issue is more comprehensive or I just misguessed at what the actual jobs were causing the hang. (I've been doing this as a 2nd monitor thing.)

I had intended to ~tail dmesg, but the right thing to run was dmesg -w and I instead did dmesg -H which is great for one-off interactive use, but was in fact not of any use in that case.

Logging into a healthy indexer we appear to not have any swap configured; I was mis-remembering that I have 8G of swap under docker, and that's because I have 8G of swap on my system, presumably due to the Ubuntu installer making a decision.

I'm going to:

• Update our clang to clang 18; this makes us consistent with the current m-c default and get us any clang enhancements that might help with memory usage.
• Update our AMI base revision in trigger-provision.py (while remaining on 22.04); this is mainly a housekeeping chore as the normal provisioning process should update to the latest 22.04 packages, but it is conceivable the base image may have some relevant changes.
• Re-provision to get us clang 18 on the indexer and that updated AMI image.
• Revert my experimental disabling of WEBGL in this push.
• Add swap initialization in mkscratch that uses the second SSD if it's present or a smallish file on the sole SSD if that's all we have. Given that it seems like the system isn't able to handle memory exhaustion without swap, this ideally helps and maybe the build can complete with just a little bit of slowdown during this weird peak utilization. I'll plan to try and log the memory statistics during the run to understand how far over we go.

Build hang output with ENABLE_WEBGL=OFF:

[ 14%] Built target bmalloc_CopyHeaders
[ 14%] Generating ../../../WTF/Headers/wtf/text/NullTextBreakIterator.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/OrdinalNumber.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringBuffer.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringBuilder.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringBuilderInternals.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringCommon.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringConcatenate.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringConcatenateNumbers.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringHash.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringHasher.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringHasherInlines.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/SuperFastHash.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/WYHash.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringImpl.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringParsingBuffer.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringSearch.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringToIntegerConversion.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/StringView.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/SymbolImpl.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/SymbolRegistry.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/TextBreakIterator.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/TextBreakIteratorInternalICU.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/TextPosition.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/TextStream.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/UTF8ConversionError.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/UniquedStringImpl.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/WTFString.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/icu/TextBreakIteratorICU.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/icu/UTextProvider.h
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/bmalloc_type.c.o
[ 14%] Generating ../../../WTF/Headers/wtf/text/icu/UTextProviderLatin1.h
[ 14%] Generating ../../../WTF/Headers/wtf/text/icu/UTextProviderUTF16.h
[ 14%] Generating ../../../WTF/Headers/wtf/threads/BinarySemaphore.h
[ 14%] Generating ../../../WTF/Headers/wtf/threads/Signals.h
[ 14%] Generating ../../../WTF/Headers/wtf/unicode/CharacterNames.h
[ 14%] Generating ../../../WTF/Headers/wtf/unicode/Collator.h
[ 14%] Generating ../../../WTF/Headers/wtf/unicode/UTF8Conversion.h
[ 14%] Generating ../../../WTF/Headers/wtf/unicode/icu/ICUHelpers.h
[ 14%] Generating ../../../WTF/Headers/wtf/linux/CurrentProcessMemoryStatus.h
[ 14%] Generating ../../../WTF/Headers/wtf/linux/ProcessMemoryFootprint.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/Application.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/ChassisType.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GMutexLocker.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GRefPtr.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GSocketMonitor.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GThreadSafeWeakPtr.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GTypedefs.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GUniquePtr.h
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Generating ../../../WTF/Headers/wtf/glib/GWeakPtr.h
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/hotbit_heap.c.o
[ 14%] Generating ../../../WTF/Headers/wtf/glib/RunLoopSourcePriority.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/Sandbox.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/SocketConnection.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/SysprofAnnotator.h
[ 14%] Generating ../../../WTF/Headers/wtf/glib/WTFGType.h
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/hotbit_heap_config.c.o
[ 14%] Generating ../../../WTF/Headers/wtf/linux/RealTimeThreads.h
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/iso_heap.c.o
[ 14%] Generating ../../../WTF/Headers/wtf/unix/UnixFileDescriptor.h
[ 14%] Built target WTF_CopyHeaders
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/iso_heap_config.c.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/iso_test_heap.c.o
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/iso_test_heap_config.c.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap.c.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 14%] Building C object Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

An interesting side-effect of the re-provisioning is that the new images end up with IMDSv2 as required which means that when retrieving instance metadata a token is required. Specifically, wget -q -O - http://instance-data/latest/meta-data/instance-id will now fail when previously it would not fail. Also it seems like maybe we should be using 169.254.169.254 instead of instance-data.

We only use the instance data once each in index.sh, rebuild-blame.sh and shell-setup.sh (because they're quasi-forks of index.sh), so this should be pretty easy to fix. Especially since it turns out we can just do ec2metadata --instance-id and not have to use the service at all.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Attachment: landed mozsearch PR for the grab bag of changes proposed

Note that as discussed in https://github.com/mozsearch/mozsearch/pull/810 I have disabled the lambda eventbridge rule "everyday-5am" that triggers config5 jobs to be initiated. One can still trigger the job manually with trigger_indexer.py if investigating fixes or trying to verify a fix, etc.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

Based on the timing of how the last job that succeeded was July 25th and the macro expansion PR landed on July 25th for bug 1583635, I'm marking this as regressed by bug 1583635. We should re-enable the lambda job when things are fixed.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

Here's an example of the ps output for a single task that was very long-running and used a bunch of resources:

ubuntu    121692  6.8 17.0 16383064 11100636 ?   D    05:34   0:32 /usr/bin/clang -DBUILDING_GTK__=1 -DBUILDING_WEBKIT=1 -DBUILDING_WITH_CMAKE=1 -DBUILDING_bmalloc -DGETTEXT_PACKAGE="WebKitGTK-6.0" -DHAVE_CONFIG_H=
1 -DJSC_GLIB_API_ENABLED -DPAS_BMALLOC=1 -D_GNU_SOURCE -I/mnt/index-scratch/wubkat/git/Source/bmalloc -I/mnt/index-scratch/wubkat/git/Source/bmalloc/bmalloc -I/mnt/index-scratch/wubkat/git/Source/bmalloc/libpas/src
/libpas -Wextra -Wall -pipe -Wno-psabi -Wno-misleading-indentation -Wno-parentheses-equality -Qunused-arguments -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wno-tautological-co
mpare -Xclang -load -Xclang /home/ubuntu/mozsearch/clang-plugin/libclang-index-plugin.so -Xclang -add-plugin -Xclang mozsearch-index -Xclang -plugin-arg-mozsearch-index -Xclang /mnt/index-scratch/wubkat/git -Xclang
 -plugin-arg-mozsearch-index -Xclang /mnt/index-scratch/wubkat/analysis -Xclang -plugin-arg-mozsearch-index -Xclang /mnt/index-scratch/wubkat/objdir -Xclang -fparse-all-comments -fno-strict-aliasing -fno-exceptions
 -ffunction-sections -fdata-sections -g -fPIC -fvisibility=hidden -Wno-missing-field-initializers -Wno-cast-align -MD -MT Source/bmalloc/CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -MF CMakeFiles/b
malloc.dir/libpas/src/libpas/jit_heap_config.c.o.d -o CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -c /mnt/index-scratch/wubkat/git/Source/bmalloc/libpas/src/libpas/jit_heap_config.c

Comment 8

[Nicolas Guichard]

Using the cmdline you provided above, I think I've tracked this to clang::format::reformat choking on the PAS_BASIC_HEAP_CONFIG_DECLARATIONS(bmalloc, BMALLOC); expansion in Source/bmalloc/libpas/src/libpas/bmalloc_heap_config.h.

Without the clang::format::reformat call in endMacroExpansion:

➜  clang-plugin git:(wip-stack) ✗ time -v sh -x ~/test.sh
+ clang -DBUILDING_GTK__=1 -DBUILDING_WEBKIT=1 -DBUILDING_WITH_CMAKE=1 -DBUILDING_bmalloc -DGETTEXT_PACKAGE=WebKitGTK-6.0 -DHAVE_CONFIG_H=1 -DJSC_GLIB_API_ENABLED -DPAS_BMALLOC=1 -D_GNU_SOURCE -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/bmalloc -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/libpas/src/libpas -Wextra -Wall -pipe -Wno-psabi -Wno-misleading-indentation -Wno-parentheses-equality -Qunused-arguments -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wno-tautological-compare -Xclang -load -Xclang /home/nico/KDAB/Customers/Mozilla/mozsearch/clang-plugin/libclang-index-plugin.so -Xclang -add-plugin -Xclang mozsearch-index -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit-analysis -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit-objdir -Xclang -fparse-all-comments -fno-strict-aliasing -fno-exceptions -ffunction-sections -fdata-sections -g -fPIC -fvisibility=hidden -Wno-missing-field-initializers -Wno-cast-align -MD -MT /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -MF /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o.d -o /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -c /home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/libpas/src/libpas/jit_heap_config.c
MOZSEARCH: /home/nico/KDAB/Customers/Mozilla/Webkit /home/nico/KDAB/Customers/Mozilla/Webkit-analysis/ /home/nico/KDAB/Customers/Mozilla/Webkit-objdir/
        Command being timed: "sh -x /home/nico/test.sh"
        User time (seconds): 3.08
...
        Maximum resident set size (kbytes): 335488 (335 MB)
...

With the clang::format::reformat call:

➜  clang-plugin git:(wip-stack) ✗ time -v sh -x ~/test.sh
+ clang -DBUILDING_GTK__=1 -DBUILDING_WEBKIT=1 -DBUILDING_WITH_CMAKE=1 -DBUILDING_bmalloc -DGETTEXT_PACKAGE=WebKitGTK-6.0 -DHAVE_CONFIG_H=1 -DJSC_GLIB_API_ENABLED -DPAS_BMALLOC=1 -D_GNU_SOURCE -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/bmalloc -I/home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/libpas/src/libpas -Wextra -Wall -pipe -Wno-psabi -Wno-misleading-indentation -Wno-parentheses-equality -Qunused-arguments -Wundef -Wpointer-arith -Wmissing-format-attribute -Wformat-security -Wcast-align -Wno-tautological-compare -Xclang -load -Xclang /home/nico/KDAB/Customers/Mozilla/mozsearch/clang-plugin/libclang-index-plugin.so -Xclang -add-plugin -Xclang mozsearch-index -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit-analysis -Xclang -plugin-arg-mozsearch-index -Xclang /home/nico/KDAB/Customers/Mozilla/Webkit-objdir -Xclang -fparse-all-comments -fno-strict-aliasing -fno-exceptions -ffunction-sections -fdata-sections -g -fPIC -fvisibility=hidden -Wno-missing-field-initializers -Wno-cast-align -MD -MT /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -MF /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o.d -o /home/nico/KDAB/Customers/Mozilla/Webkit-CMakeFiles/bmalloc.dir/libpas/src/libpas/jit_heap_config.c.o -c /home/nico/KDAB/Customers/Mozilla/Webkit/Source/bmalloc/libpas/src/libpas/jit_heap_config.c
MOZSEARCH: /home/nico/KDAB/Customers/Mozilla/Webkit /home/nico/KDAB/Customers/Mozilla/Webkit-analysis/ /home/nico/KDAB/Customers/Mozilla/Webkit-objdir/
        Command being timed: "sh -x /home/nico/test.sh"
        User time (seconds): 138.16
...
        Maximum resident set size (kbytes): 16260360 (16.2 GB !!)
...

Comment 9

[Andrew Sutherland [:asuth] (he/him)]

Thank you for digging into that! There definitely does seem to be a bit of a resource utilization difference! I'll spin up try runs for config1 and config5 and assuming there isn't a huge difference for config1, I presume we should land this.

Comment 10

[Andrew Sutherland [:asuth] (he/him)]

Unfortunately I ended up not having the spare time I'd hoped to have during/around MozWeek.

Comment 11

[Nicolas Guichard]

I've diagnosed this further, the culprit really is ColumnLimit, when setting ColumnLimit to 0, I get much reasonable memory usage:

➜  clang-plugin git:(wip-stack) ✗ time -v sh -x ~/test.sh
User time (seconds): 53.82
Maximum resident set size (kbytes): 351292 (350 MB)

The time spent on reformatting for that file is still very high (54 seconds with, 3 seconds without) but memory usage is back to normal. I'll launch a config5 indexer with ColumnLimit = 0.

Comment 12

[Andrew Sutherland [:asuth] (he/him)]

Awesome! Thank you! The good news is the diagram of the include uses of bmalloc_heap_config.h suggests that we'll only be taking the performance hit for a limited number of files. (Object.keys(SYM_INFO).length returns 34.)

Comment 13

[Nicolas Guichard]

Attachment: Mozsearch Pull Request

Comment 14

[Nicolas Guichard]

Attachment: Mozsearch-Mozilla Pull Request

Wubkat build failed, trying again with a new dependency.

Comment 15

[Nicolas Guichard]

Attachment: Bug 1912078 - Don't reformat huge macro expansions. r=asuth!

This disables reformating macros that are longer than a given threshold
as they take too long to reformat (and too much memory when
ColumnLimit != 0).

The threshold is lower when the macro expansion in located in a file
that is likely to be included from multiple TUs.

See https://github.com/mozsearch/mozsearch/pull/815

Comment 16

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ec1e50269e4b
Don't reformat huge macro expansions. r=asuth

Comment 17

[Sandor Molnar[:smolnar]]

https://hg.mozilla.org/mozilla-central/rev/ec1e50269e4b