1658040 - `mach package-multi-locale` busted locally

Status: RESOLVED FIXED

(Firefox Build System :: Android Studio and Gradle Integration, defect)

Description

[James Willcox (:snorp) ([e-mail]) (he/him)]

Trying to build a local multi-locale GeckoView fails at the mach package-multi-locale step with:

Task :machStagePackageForGeckoview FAILED
Executing task :machStagePackageForGeckoview
:machStagePackageForGeckoview> gmake: Entering directory '/home/snorp/source/objdirs/objdir-android-opt/mobile/android/installer'
:machStagePackageForGeckoview> /home/snorp/source/objdirs/objdir-android-opt/_virtualenvs/init_py3/bin/python -m mozbuild.action.file_generate /home/snorp/source/gecko/toolkit/locales/gen_multilocale.py main '../../../dist/bin/res/multilocale.txt' .deps/multilocale.txt.pp '../../../dist/bin/res/multilocale.txt' de en-US it
:machStagePackageForGeckoview> printf '\n[multilocale]\n' > locale-manifest.in
:machStagePackageForGeckoview> printf '@BINPATH@/res/multilocale.txt\n' >> locale-manifest.in
:machStagePackageForGeckoview> for LOCALE in de en-US it ;
:machStagePackageForGeckoview> do
:machStagePackageForGeckoview> for ENTRY in @BINPATH@/chrome/ ;
:machStagePackageForGeckoview> do
:machStagePackageForGeckoview> printf "$ENTRY""$LOCALE"'@JAREXT@\n' >> locale-manifest.in;
:machStagePackageForGeckoview> printf "$ENTRY""$LOCALE"'.manifest\n' >> locale-manifest.in;
:machStagePackageForGeckoview> done
:machStagePackageForGeckoview> done
:machStagePackageForGeckoview> NO_PKG_FILES="core bsdecho js js-config jscpucfg nsinstall viewer TestGtkEmbed elf-dynstr-gc mangle* maptsv* mfc* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* certutil* pk12util* BadCertAndPinningServer* DelegatedCredentialsServer* OCSPStaplingServer* SanctionsTestServer* GenerateOCSPResponse* chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests .dSYM "
:machStagePackageForGeckoview> /home/snorp/source/objdirs/objdir-android-opt/virtualenvs/init_py3/bin/python /home/snorp/source/gecko/toolkit/mozapps/installer/packager.py -DPKG_LOCALE_MANIFEST=/home/snorp/source/objdirs/objdir-android-opt/mobile/android/installer/locale-manifest.in -DMOZ_APP_NAME=fennec -DPREF_DIR=defaults/pref -DJAREXT= -DMOZ_CHILD_PROCESS_NAME=libplugin-container.so -DANDROID_CPU_ARCH=x86_64 -DMOZ_DEBUG=1 -DMOZ_ANDROID_EXCLUDE_FONTS=1 -DMOZ_GECKOVIEW_JAR=1 -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DDIR_MACOS= -DDIR_RESOURCES= -DMOZ_FOLD_LIBS=1 -DBINPATH=bin -DENABLE_MARIONETTE=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DANDROID=1 -DBROWSER_CHROME_URL=chrome://browser/content/browser.xul -DBROWSER_CHROME_URL_QUOTED='"chrome://browser/content/browser.xul"' -DBUILD_CTYPES=1 -DCROSS_COMPILE=1 -DEARLY_BETA_OR_EARLIER=1 -DENABLE_SHARED_MEMORY=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DENABLE_WASM_BULKMEM_OPS=1 -DENABLE_WASM_CRANELIFT=1 -DENABLE_WASM_GC=1 -DENABLE_WASM_MULTI_VALUE=1 -DENABLE_WASM_REFTYPES=1 -DENABLE_WASM_SIMD=1 -DFORCE_PR_LOG=1 -DGTEST_HAS_CLONE=0 -DGTEST_HAS_RTTI=0 -DGTEST_OS_LINUX_ANDROID=1 -DGTEST_USE_OWN_TR1_TUPLE=1 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_ARC4RANDOM=1 -DHAVE_ARC4RANDOM_BUF=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FTS_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GETTID=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX
:machStagePackageForGeckoview> PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DJS_64BIT=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_GC_SMALL_CHUNK_SIZE=1 -DJS_GC_ZEAL=1 -DJS_HAS_INTL_API=1 -DJS_HAS_TYPED_OBJECTS=1 -DJS_JITSPEW=1 -DJS_MASM_VERBOSE=1 -DJS_PUNBOX64=1 -DJS_STRUCTURED_SPEW=1 -DJS_TRACE_LOGGING=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR=const -DMOZILLA_UAVERSION='"80.0"' -DMOZILLA_VERSION='"80.0a1"' -DMOZILLA_VERSION_U=80.0a1 -DMOZ_ANDROID_HISTORY=1 -DMOZ_APP_UA_NAME='"Firefox"' -DMOZ_APP_UA_VERSION='"80.0a1"' -DMOZ_AV1=1 -DMOZ_BUILD_APP=mobile/android -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DAV1D_ASM=1 -DMOZ_DEBUG=1 -DMOZ_DEBUG_RUST=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_PREFIX='"lib"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_DUMP_PAINTING=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GECKO_PROFILER_PARSE_ELF=1 -DMOZ_GFX_OPTIMIZE_MOBILE=1 -DMOZ_IGNORE_PAINT_WILL_RESAMPLE=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LAYOUT_DEBUGGER=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LINKER=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.fennecsnorpdebug -DMOZ_MEMORY=1 -DMOZ_NEW_CERT_STORAGE=1 -DMOZ_NEW_NOTIFICATION_STORE=1 -DMOZ_NEW_XULSTORE=1 -DMOZ_NO_SMART_CARDS=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PROFILER_MEMORY=1 -DMOZ_PROFILING=1 -DMOZ_RAW=1 -DMOZ_REFLOW_PERF=1 -DMOZ_REFLOW_PERF_DSP=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_R
:machStagePackageForGeckoview> UST_SIMD=1 -DMOZ_SAMPLE_TYPE_S16=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_FREETYPE=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_TREMOR=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_USE_NATIVE_POPUP_WINDOWS=1 -DMOZ_VERIFY_MAR_SIGNATURE=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_ANDROID=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DWASM_CODEGEN_DEBUG=1 -DWASM_PRIVATE_REFTYPES=1 -DXP_LINUX=1 -DXP_UNIX=1 -DX_DISPLAY_MISSING=1 -DZLIB_IN_MOZGLUE=1 -D_LIBCPP_ALWAYS_INLINE='' -D_LIBCPP_HIDE_FROM_ABI='attribute((exclude_from_explicit_instantiation))' -D_LIBCPP_INLINE_VISIBILITY='' -D_REENTRANT=1 -DAB_CD=en-US
:machStagePackageForGeckoview> --format omni
:machStagePackageForGeckoview> --removals /home/snorp/source/gecko/mobile/android/installer/removed-files.in
:machStagePackageForGeckoview> --ignore-errors
:machStagePackageForGeckoview> --ignore-broken-symlinks
:machStagePackageForGeckoview> --minify
:machStagePackageForGeckoview>
:machStagePackageForGeckoview>
:machStagePackageForGeckoview>
:machStagePackageForGeckoview> /home/snorp/source/gecko/mobile/android/installer/package-manifest.in '../../../dist' '../../../dist'/geckoview
:machStagePackageForGeckoview>
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:23: Missing file(s): bin/dictionaries/
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:88: Missing file(s): bin/package-name.txt
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:99: Missing file(s): bin/components/components.manifest
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:134: Missing file(s): bin/features/*
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:154: Missing file(s): bin/defaults/pref/channel-prefs.js
:machStagePackageForGeckoview> Warning: $SRCDIR/mobile/android/installer/package-manifest.in:210: Missing file(s): bin/crashreporter-override.ini
:machStagePackageForGeckoview> Error: "chrome/chrome.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/devtools-startup.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/devtools.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/en-US.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/geckoview.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/marionette.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/pdfjs.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/recording.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Error: "chrome/toolkit.manifest" is included from "chrome.manifest", which is outside "chrome"
:machStagePackageForGeckoview> Traceback (most recent call last):
:machStagePackageForGeckoview> File "/home/snorp/source/gecko/toolkit/mozapps/installer/packager.py", line 249, in <module>
:machStagePackageForGeckoview> main()
:machStagePackageForGeckoview> File "/home/snorp/source/gecko/toolkit/mozapps/installer/packager.py", line 208, in main
:machStagePackageForGeckoview> copier.add(mozpath.join(respath, 'removed-files'), removals)
:machStagePackageForGeckoview> File "/usr/lib64/python3.8/contextlib.py", line 120, in exit
:machStagePackageForGeckoview> next(self.gen)
:machStagePackageForGeckoview> File "/home/snorp/source/gecko/python/mozbuild/mozpack/errors.py", line 131, in accumulate
:machStagePackageForGeckoview> raise AccumulatedErrors()
:machStagePackageForGeckoview> mozpack.errors.AccumulatedErrors
:machStagePackageForGeckoview> gmake: *** [/home/snorp/source/gecko/toolkit/mozapps/installer/packager.mk:25: stage-package] Error 1
:machStagePackageForGeckoview> gmake: *** [/home/snorp/source/gecko/toolkit/mozapps/installer/packager.mk:25: stage-package] Error 1gmake: Leaving directory '/home/snorp/source/objdirs/objdir-android-opt/mobile/android/installer'

FAILURE: Build failed with an exception.

• What went wrong:
  Execution failed for task ':machStagePackageForGeckoview'.

Process 'command '/usr/bin/gmake'' finished with non-zero exit value 2

Comment 1

[James Willcox (:snorp) ([e-mail]) (he/him)]

Called/Axel any ideas about this? I'm trying to get this going locally in order to debug Bug 1656515.

Comment 2

[Nick Alexander :nalexander [he/him]]

It's worth noting that this doesn't fail for me with an artifact build using ./mach package-multi-locale --locales en-US de. However, the resulting omnijar does not include chrome/de/**. (It does include a res/multilocale.txt with de,en-US.)

Comment 3

[Axel Hecht [:Pike]]

From what I remember last time I tried to understand the faster-build backend, the jar.mn logic can't deal with locale-dependent paths.

So I'd expect artifact builds to not work.

No idea about the error messages snorp sees.

Comment 4

[Nick Alexander :nalexander [he/him]]

(In reply to Axel Hecht [:Pike] from comment #3)

From what I remember last time I tried to understand the faster-build backend, the jar.mn logic can't deal with locale-dependent paths.

So I'd expect artifact builds to not work.

Oh, of course! I forgot about the build-backend hijinks needed to make this work with artifact builds! Thanks for the nudge, Axel. I will try to look at this again.

Comment 5

[Nick Alexander :nalexander [he/him]]

I dug into this today, and figured out why --disable-compile-environment builds were broken, reproed the errors snorp was witnessing, and figured out why they were happening. tl;dr: set GRADLE_INVOKED_WITHIN_MACH_BUILD=1 in the environment before running mach package-multi-locale ....

I'll post some patches shortly.

Comment 6

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1658040 - Part 1: Fix local `mach package-multi-locale ...` locally. r?snorp

This succeeded in automation because we check for MOZILLA_OFFICIAL
before invoking mach from within
Gradle.
Locally, the check for AB_CD=multi in the environment used to catch
this but it's no longer used. Rather than continue confusion with
AB_CD=multi make ... versus make AB_CD=multi ..., we'll use the
Gradle-specific environment variable.

Comment 7

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1658040 - Part 2: Deny `mach package-multi-locale ...` when it's going to fail. r?#build

Depends on D87331

Comment 8

[Nick Alexander :nalexander [he/him]]

Attachment: Bug 1658040 - Part 3: Allow `mach build` with `--disable-compile-environment`. r?#build

Builds without a compilation environment don't produce
libxul_so.list, and therefore this GeneratedFile can't depend on
it. But this GeneratedFile only makes sense for compilation, so we
just make it conditional.

Depends on D87332

Comment 9

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ed62fe310c28
Part 1: Fix local `mach package-multi-locale ...` locally. r=snorp
https://hg.mozilla.org/integration/autoland/rev/2797a177bc6d
Part 2: Deny `mach package-multi-locale ...` when it's going to fail. r=froydnj
https://hg.mozilla.org/integration/autoland/rev/fb7828bba1dd
Part 3: Allow `mach build` with `--disable-compile-environment`. r=froydnj

Comment 10

[Cosmin Sabou [:CosminS]]

https://hg.mozilla.org/mozilla-central/rev/ed62fe310c28
https://hg.mozilla.org/mozilla-central/rev/2797a177bc6d
https://hg.mozilla.org/mozilla-central/rev/fb7828bba1dd