1749385 - Build dependencies broken by `GeckoProcessTypes_CrashManager_*`

Status: RESOLVED FIXED

(Core :: Security: Process Sandboxing, defect, P2)

Description

[Nick Alexander :nalexander [he/him]]

I believe that Bug 1726804 has broken build dependencies, possibly only for artifact builds. When I build -j1, the build manifest with CrashManager.jsm is processed before any file generation takes place. With the #includesubst approach, I don't really see how the build dependencies can be determined. Perhaps something else is interfering with the .pp/deps mechanism that handles dependencies and that should be catching this case?

In general, we should strongly avoid preprocessing JavaScript source code in this manner. I would much prefer that we use the strong dynamic facilities in JS to arrange whatever is wanted here (stitching objects together, AFAICT) at run-time rather than doing this at build-time.

Comment 1

[Nick Alexander :nalexander [he/him]]

Error looks like:

/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact-de/_virtualenvs/build/bin/python -m mozbuild.action.process_install_manifest \
		 \
		--track install_dist_bin.track \
		/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact-de/dist/bin \
		-DAB_CD=en-US \
		-DA11Y_LOG=1 -DACCESSIBILITY=1 -DBROWSER_CHROME_URL=chrome://browser/content/browser.xhtml -DBROWSER_CHROME_URL_QUOTED='"chrome://browser/content/browser.xhtml"' -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_SHARED_MEMORY=1 -DENABLE_SPIDERMONKEY_TELEMETRY=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DENABLE_WASM_CALL_INDIRECT_NULL=1 -DENABLE_WASM_EXCEPTIONS=1 -DENABLE_WASM_EXTENDED_CONST=1 -DENABLE_WASM_FUNCTION_REFERENCES=1 -DENABLE_WASM_GC=1 -DENABLE_WASM_MEMORY64=1 -DENABLE_WASM_MOZ_INTGEMM=1 -DENABLE_WASM_RELAXED_SIMD=1 -DENABLE_WASM_SIMD=1 -DENABLE_WASM_SIMD_WORMHOLE=1 -DENABLE_WASM_TYPE_REFLECTIONS=1 -DENABLE_WEBDRIVER=1 -DFORCE_PR_LOG=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_FORCEINLINE=1 -DHAVE_LOCALECONV=1 -DJS_64BIT=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_GC_ZEAL=1 -DJS_HAS_CTYPES=1 -DJS_HAS_INTL_API=1 -DJS_JITSPEW=1 -DJS_MASM_VERBOSE=1 -DJS_PUNBOX64=1 -DJS_STRUCTURED_SPEW=1 -DJS_TRACE_LOGGING=1 -DMALLOC_H='<malloc.h>' -DMOZILLA_UAVERSION='"97.0"' -DMOZILLA_VERSION='"97.0a1"' -DMOZILLA_VERSION_U=97.0a1 -DMOZJS_MAJOR_VERSION=97 -DMOZJS_MINOR_VERSION=0 -DMOZ_AARCH64_JSCVT=0 -DMOZ_APPLEMEDIA=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"97.0a1"' -DMOZ_AUDIOUNIT_RUST=1 -DMOZ_AUTH_EXTENSION=1 -DMOZ_AV1=1 -DMOZ_BLOCK_PROFILE_DOWNGRADE=1 -DMOZ_BUILD_APP=browser -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DEBUG=1 -DMOZ_DEDICATED_PROFILES=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_PREFIX='"lib"' -DMOZ_DLL_SUFFIX='".dylib"' -DMOZ_DMD=1 -DMOZ_DUMP_PAINTING=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_HAS_REMOTE=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_JXL=1 -DMOZ_LAYOUT_DEBUGGER=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightlydebug -DMOZ_NEW_NOTIFICATION_STORE=1 -DMOZ_NEW_XULSTORE=1 -DMOZ_NORMANDY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PLACES=1 -DMOZ_PROFILER_MEMORY=1 -DMOZ_PROFILING=1 -DMOZ_PROXY_DIRECT_FAILOVER=1 -DMOZ_RAW=1 -DMOZ_REFLOW_PERF=1 -DMOZ_REFLOW_PERF_DSP=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SANDBOX=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USING_WASM_SANDBOXING=1 -DMOZ_VERIFY_MAR_SIGNATURE=1 -DMOZ_VORBIS=1 -DMOZ_WASM_SANDBOXING_EXPAT=1 -DMOZ_WASM_SANDBOXING_GRAPHITE=1 -DMOZ_WASM_SANDBOXING_HUNSPELL=1 -DMOZ_WASM_SANDBOXING_OGG=1 -DMOZ_WASM_SANDBOXING_WOFF2=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_COCOA=1 -DNECKO_WIFI=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_BUILD_REFCNT_LOGGING=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DWASM_CODEGEN_DEBUG=1 -DWASM_PRIVATE_REFTYPES=1 -DXP_DARWIN=1 -DXP_MACOSX=1 -DXP_UNIX=1 \
		install_dist_bin
Traceback (most recent call last):
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 863, in do_include
    args = io.open(args, "rU", encoding="utf-8")
FileNotFoundError: [Errno 2] No such file or directory: '/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact-de/toolkit/components/crashes/GeckoProcessTypes_CrashManager_map.js'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/Cellar/python@3.9/3.9.9/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py", line 133, in <module>
    log_build_task(main, sys.argv[1:])
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/util.py", line 18, in log_build_task
    return f(*args, **kwargs)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py", line 110, in main
    result = process_manifest(
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/action/process_install_manifest.py", line 67, in process_manifest
    result = copier.copy(
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozpack/copier.py", line 434, in copy
    copy_results.append((destfile, f.copy(destfile, skip_if_older)))
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozpack/files.py", line 631, in copy
    pp.processFile(input=input, output=dest, depfile=deps_out)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 435, in processFile
    self.do_include(input, False)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 896, in do_include
    self.handleLine(l)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 654, in handleLine
    cmd(args)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 908, in do_includesubst
    self.do_include(args)
  File "/Users/nalexander/Mozilla/gecko/python/mozbuild/mozbuild/preprocessor.py", line 867, in do_include
    raise Preprocessor.Error(self, "FILE_NOT_FOUND", _to_text(args))
mozbuild.preprocessor.Preprocessor.Error: ('$SRCDIR/toolkit/components/crashes/CrashManager.jsm', 170, 'FILE_NOT_FOUND', '/Users/nalexander/Mozilla/objdirs/objdir-browser-artifact-de/toolkit/components/crashes/GeckoProcessTypes_CrashManager_map.js')

Comment 2

[Nick Alexander :nalexander [he/him]]

glandium: is this supposed to work? Are #includesubst blocks in EXTRA_PP_JS_MODULES expected to work in artifact builds? Maybe there's a tiering issue where the included GeneratedFile items are supposed to be built before the manifests are processed?

Comment 3

[Mike Hommey [:glandium]]

Yeah, there's a missing implicit dependency in the "faster" make backend, and not an easy one to fix other than with manual dependencies :-/

Comment 4

[BugBot [:suhaib / :marco/ :calixte]]

Set release status flags based on info from the regressing bug 1726804

Comment 5

[:gerard-majax]

Attachment: Bug 1749385 - Generatate CrashManager.jsm from template r?gsvelto!

Comment 6

[:gerard-majax]

Attachment: Bug 1749385 - Generate CrashManager.jsm from template r?gsvelto!

Comment 7

[:gerard-majax]

I believe the attach patch should fix the issue as suggested by :glandium on Matrix, can you verify on your side? Thanks!

Comment 8

[Nick Alexander :nalexander [he/him]]

(In reply to Alexandre LISSY :gerard-majax from comment #7)

I believe the attach patch should fix the issue as suggested by :glandium on Matrix, can you verify on your side? Thanks!

Verified! Thanks for jumping on this so quickly!

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4c0498f12793
Generate CrashManager.jsm from template r=gsvelto,nalexander

Comment 10

[Cristina Cozmuta (:CrissCozmuta)]

https://hg.mozilla.org/mozilla-central/rev/4c0498f12793

Comment 11

[BugBot [:suhaib / :marco/ :calixte]]

The patch landed in nightly and beta is affected.
:gerard-majax, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Comment 12

[:gerard-majax]

I dont think we need an uplift for that, even if it's not super risky