1923545 - config5 wubkat indexer failing on building skia mysteriously

Status: REOPENED

(Webtools :: Searchfox, task)

Description

[Andrew Sutherland [:asuth] (he/him)]

As a follow-up to bug 1912078 (and with the PR happening notionally associated with that bug), I made the changes in https://github.com/mozsearch/mozsearch-mozilla/pull/264 and landed them concurrently with the changes in bug 1922408 to update us to Ubuntu 24.04.

Failure emails look like:

[ 28%] Building CXX object Source/ThirdParty/skia/CMakeFiles/Skia.dir/modules/skcms/src/skcms_TransformBaseline.cc.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 28%] Building CXX object Source/ThirdParty/skia/CMakeFiles/Skia.dir/modules/skcms/src/skcms_TransformHsw.cc.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 28%] Building CXX object Source/ThirdParty/skia/CMakeFiles/Skia.dir/modules/skcms/src/skcms_TransformSkx.cc.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 28%] Linking CXX static library ../../../lib/libSkia.a
[ 28%] Built target Skia
make: *** [Makefile:166: all] Error 2

Logging into the machine and minimally doing export VERBOSE=1 (make VERBOSE=1 -j16 should also work) didn't provide any extra info. Running make in debug mode also wasn't particularly informative. I briefly tried to disable skia but that didn't seem like an actual option at this point. I didn't dig in any further than that due to higher priority tasks.

I'm going to revert https://github.com/mozsearch/mozsearch-mozilla/pull/264 so we can see what happens tomorrow; it's conceivable developer-mode changes something. That will re-enable the docgen stuff which we don't think we want, but is fine for isolating what's going on.

Comment 1

[Andrew Sutherland [:asuth] (he/him)]

Attachment: landed PR 264 that was subsequently reverted by 265

Comment 2

[Andrew Sutherland [:asuth] (he/him)]

Attachment: landed PR 265 which is a reversion of 264

Comment 3

[Andrew Sutherland [:asuth] (he/him)]

Reverting fixed things. In the successful run we saw some warnings like:

 WARN logged_span: tools::file_format::repo_data_ingestion: Problem gathering metadata for /mnt/index-scratch/wubkat/objdir/CMakeFiles/CMakeScratch/TryCompile-j69lTk/CheckSymbolExists.c: No such file or directory (os error 2) name="repo_ingestion" uuid=75dee058-08d5-45ce-b6d0-a82e85859afb
 WARN logged_span: tools::file_format::repo_data_ingestion: Problem gathering metadata for /mnt/index-scratch/wubkat/objdir/CMakeFiles/CMakeScratch/TryCompile-SjlyHg/src.cxx: No such file or directory (os error 2) name="repo_ingestion" uuid=75dee058-08d5-45ce-b6d0-a82e85859afb
 WARN logged_span: tools::file_format::repo_data_ingestion: Problem gathering metadata for /mnt/index-scratch/wubkat/objdir/CMakeFiles/CMakeScratch/TryCompile-L7nRyu/src.c: No such file or directory (os error 2) name="repo_ingestion" uuid=75dee058-08d5-45ce-b6d0-a82e85859afb
 WARN logged_span: tools::file_format::repo_data_ingestion: Problem gathering metadata for /mnt/index-scratch/wubkat/objdir/CMakeFiles/CMakeScratch/TryCompile-MHdY1L/CheckFunctionExists.c: No such file or directory (os error 2) name="repo_ingestion" uuid=75dee058-08d5-45ce-b6d0-a82e85859afb

We already have a mechanism to turn these into "info"s that we're already using for wubkat: https://searchfox.org/mozsearch-mozilla/rev/9a6b6f1e3690a83b23b3a1ec21ec5ca61d87f9b0/config5.json#18-20

"ignore_missing_path_prefixes": [
  "$WORKING/wubkat/objdir/CMakeFiles/CMakeTmp/"
],

Per https://cmake.org/cmake/help/latest/command/try_compile.html#id5 it seems like cmake 3.25 changed CMakeTmp to CMakeScratch so it seems like I should just change the path.

In the commit for that I'll try disabling the documentation again (but without switching us to developer mode from disabling the gpu process) and we'll see how that works tomorrow.

Comment 4

[Andrew Sutherland [:asuth] (he/him)]

Attachment: Landed mozsearch-mozilla PR to quiet the warnings and re-disable docgen

The cmake warnings are now quieted but there is a warning coming from the dynamic package installation that I'm looking at; some discussion on the config5 warning email threads.

Comment 5

[Andrew Sutherland [:asuth] (he/him)]

The re-provisioning (related to bug 1924227) seems to have successfully quieted the config5 warnings.

Comment 6

[Andrew Sutherland [:asuth] (he/him)]

skia is failing again for the past 2 days, roughly like so:

[ 29%] Building CXX object Source/ThirdParty/skia/CMakeFiles/Skia.dir/modules/svg/src/SkSVGTransformableNode.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 29%] Building CXX object Source/ThirdParty/skia/CMakeFiles/Skia.dir/modules/svg/src/SkSVGUse.cpp.o
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
MOZSEARCH: /mnt/index-scratch/wubkat/git /mnt/index-scratch/wubkat/analysis/ /mnt/index-scratch/wubkat/objdir/
[ 29%] Linking CXX static library ../../../lib/libSkia.a
[ 29%] Built target Skia
make: *** [Makefile:156: all] Error 2
+ handle_tree_error 'tree build script'
+ local 'msg=tree build script'

Comment 7

[Andrew Sutherland [:asuth] (he/him)]

It seems like the problem went away again on Feb 8th and today's (Feb 10th) index seems legit. I'm going to leave this open for now but we can close again if the problem doesn't show up again for a while or we figure out what might have changed.