Closed Bug 1504489 Opened 6 years ago Closed 6 years ago

Perma-failing Bocf make[1]: *** [automation/package-tests] Error 2

Categories

(Firefox Build System :: General, defect, P5)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(firefox65 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla65
Tracking Flags:
Tracking Status
firefox65 --- fixed

People

(Reporter: nataliaCs, Assigned: marco)

References

Details

(Keywords: intermittent-failure)

Attachments

(2 files, 1 obsolete file)

Compress debug info section to avoid package failure due to too big libxul.so
843 bytes, patch
marco
: review+
Details | Diff | Splinter Review
Make fuzzing ccov mozconfig import the code coverage opt mozconfig
1.47 KB, patch
decoder
: review+
Details | Diff | Splinter Review 
+++ This bug was initially created as a clone of Bug #1484073 +++

Push : https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&searchStr=linux%2Cx64%2Copt%2Cbuild-linux64-fuzzing-ccov%2Fopt%2C%28bocf%29&revision=50bad93c5bdf9b1e8bae884b895f6b9da9a21a48&selectedJob=209638287

Failure log: https://treeherder.mozilla.org/logviewer.html#?job_id=209638287&repo=mozilla-central&lineNumber=42310

task 2018-11-03T19:25:43.589Z] 19:25:43     INFO -  package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-11-03T19:25:43.589Z] 19:25:43     INFO -  /builds/worker/workspace/build/src/build/moz-automation.mk:84: recipe for target 'automation/package-tests' failed
[task 2018-11-03T19:25:43.589Z] 19:25:43     INFO -  make[1]: *** [automation/package-tests] Error 2
[task 2018-11-03T19:25:43.589Z] 19:25:43     INFO -  make[1]: *** Waiting for unfinished jobs....
[task 2018-11-03T19:29:32.666Z] 19:29:32     INFO -  package> make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/browser/installer'
[task 2018-11-03T19:29:32.666Z] 19:29:32     INFO -  package> OMNIJAR_NAME=omni.ja \
[task 2018-11-03T19:29:32.667Z] 19:29:32     INFO -  package> NO_PKG_FILES="core bsdecho js js-config jscpucfg nsinstall viewer TestGtkEmbed elf-dynstr-gc mangle* maptsv* mfc* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* certutil* pk12util* BadCertServer* OCSPStaplingServer* SymantecSanctionsServer* GenerateOCSPResponse* chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM " \
[task 2018-11-03T19:29:32.670Z] 19:29:32     INFO -  package> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python /builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.py -DPKG_LOCALE_MANIFEST=/builds/worker/workspace/build/src/obj-firefox/browser/installer/locale-manifest.in -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/preferences -DMOZ_GTK=1 -DMOZ_GTK3=1 -DJAREXT= -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DNECKO_WIFI -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DDIR_MACOS= -DDIR_RESOURCES= -DBINPATH='bin' -DRESPATH='bin' -DLPROJ_ROOT=en -DENABLE_MARIONETTE=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=4 -DATK_REV_VERSION=0 -DBROWSER_CHROME_URL=chrome://browser/content/browser.xul -DBROWSER_CHROME_URL_QUOTED='"chrome://browser/content/browser.xul"' -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DENABLE_WASM_CRANELIFT=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=65.0a1 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DFUZZING=1 -DGDK_VERSION_MAX_ALLOWED=GDK_VERSION_3_4 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_64BIT=1 -DJS_BUILD_BINAST=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_PUNBOX64=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_OFFICIAL=1 -DMOZILLA_UAVERSION='"65.0"' -DMOZILLA_VERSION='"65.0a1"' -DMOZILLA_VERSION_U=65.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ALLOW_LEGACY_EXTENSIONS=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"65.0a1"' -DMOZ_AV1=1 -DMOZ_BLOCK_PROFILE_DOWNGRADE=1 -DMOZ_BUILD_APP=browser -DMOZ_BUILD_WEBRENDER=1 -DMOZ_BUNDLED_FONTS=1 -DMOZ_CODE_COVERAGE=1 -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DEDICATED_PROFILES=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_PREFIX='"lib"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightly -DMOZ_MEMORY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PROFILER_MEMORY=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_RUST_SIMD=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TELEMETRY_ON_BY_DEFAULT=1 -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VERIFY_MAR_SIGNATURE=1 -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DAB_CD=en-US \
[task 2018-11-03T19:29:32.671Z] 19:29:32     INFO -  package> 	--format omni \
[task 2018-11-03T19:29:32.671Z] 19:29:32     INFO -  package> 	--removals /builds/worker/workspace/build/src/browser/installer/removed-files.in \
[task 2018-11-03T19:29:32.672Z] 19:29:32     INFO -  package> 	 \
[task 2018-11-03T19:29:32.672Z] 19:29:32     INFO -  package> 	--minify \
[task 2018-11-03T19:29:32.672Z] 19:29:32     INFO -  package> 	 \
[task 2018-11-03T19:29:32.673Z] 19:29:32     INFO -  package> 	 \
[task 2018-11-03T19:29:32.673Z] 19:29:32     INFO -  package> 	--optimizejars \
[task 2018-11-03T19:29:32.673Z] 19:29:32     INFO -  package> 	--compress none \
[task 2018-11-03T19:29:32.674Z] 19:29:32     INFO -  package> 	/builds/worker/workspace/build/src/browser/installer/package-manifest.in '../../dist' '../../dist'/firefox \
[task 2018-11-03T19:29:32.674Z] 19:29:32     INFO -  package>
[task 2018-11-03T19:29:32.674Z] 19:29:32     INFO -  package> Executing /builds/worker/workspace/build/src/obj-firefox/dist/bin/shlibsign -v -o ../../dist/firefox/libsoftokn3.chk -i ../../dist/firefox/libsoftokn3.so
[task 2018-11-03T19:29:32.675Z] 19:29:32     INFO -  package> Library File: ../../dist/firefox/libsoftokn3.so 1607592 bytes
[task 2018-11-03T19:29:32.675Z] 19:29:32     INFO -  package> Check File: ../../dist/firefox/libsoftokn3.chk
[task 2018-11-03T19:29:32.675Z] 19:29:32     INFO -  package>   hash: 32 bytes
[task 2018-11-03T19:29:32.676Z] 19:29:32     INFO -  package>     9e 6b 6a 0a 74 9f 17 88 3c 4a
[task 2018-11-03T19:29:32.676Z] 19:29:32     INFO -  package>     72 01 c5 ce 1a 2a 41 43 6c b5
[task 2018-11-03T19:29:32.676Z] 19:29:32     INFO -  package>     15 cb c1 ad 42 de 27 a1 c0 94
[task 2018-11-03T19:29:32.677Z] 19:29:32     INFO -  package>     7b ff
[task 2018-11-03T19:29:32.677Z] 19:29:32     INFO -  package>   signature: 64 bytes
[task 2018-11-03T19:29:32.677Z] 19:29:32     INFO -  package>     0d 63 b5 42 6c 8e d9 1c fd fe
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     15 08 6e 6a 6a 04 6a 25 0e da
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     0e d1 4e b6 11 40 e1 a5 16 5d
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     4c 22 1b 2f c7 27 5e 59 96 e4
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     af 0a 4d 1b db d2 60 08 e9 c6
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     ab 75 d6 e6 13 80 4e 1f 08 d7
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package>     a6 40 2d 5b
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package> moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
[task 2018-11-03T19:29:32.678Z] 19:29:32     INFO -  package> Generate a DSA key pair ...
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package> Executing /builds/worker/workspace/build/src/obj-firefox/dist/bin/shlibsign -v -o ../../dist/firefox/libnssdbm3.chk -i ../../dist/firefox/libnssdbm3.so
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package> Library File: ../../dist/firefox/libnssdbm3.so 1231848 bytes
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package> Check File: ../../dist/firefox/libnssdbm3.chk
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package>   hash: 32 bytes
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package>     82 3c 76 42 f6 9b 47 d9 1e 8d
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package>     d1 2c d6 35 4d b9 6e d1 9b ab
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package>     6a 22 30 c4 cc e9 56 c8 49 e2
[task 2018-11-03T19:29:32.679Z] 19:29:32     INFO -  package>     0f 77
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>   signature: 64 bytes
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     00 1c 0b d3 d1 50 09 ee 6e 98
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     28 56 88 c6 a9 11 ef 5f 7d 32
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     ad d3 ce e7 9d 67 b0 a1 df f1
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     04 10 71 16 ea f7 c7 98 b6 b3
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     9a 9e 6d 11 48 97 38 80 4e 61
[task 2018-11-03T19:29:32.680Z] 19:29:32     INFO -  package>     f4 12 1f e4 8d 48 68 30 b4 32
[task 2018-11-03T19:29:32.681Z] 19:29:32     INFO -  package>     31 33 db 5f
[task 2018-11-03T19:29:32.681Z] 19:29:32     INFO -  package> moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB
[task 2018-11-03T19:29:32.681Z] 19:29:32     INFO -  package> Generate a DSA key pair ...
[task 2018-11-03T19:29:32.681Z] 19:29:32     INFO -  package> Executing /builds/worker/workspace/build/src/obj-firefox/dist/bin/shlibsign -v -o ../../dist/firefox/libfreeblpriv3.chk -i ../../dist/firefox/libfreeblpriv3.so
[task 2018-11-03T19:29:32.682Z] 19:29:32     INFO -  package> Library File: ../../dist/firefox/libfreeblpriv3.so 2719808 bytes
[task 2018-11-03T19:29:32.682Z] 19:29:32     INFO -  package> Check File: ../../dist/firefox/libfreeblpriv3.chk
[task 2018-11-03T19:29:32.682Z] 19:29:32     INFO -  package>   hash: 32 bytes
[task 2018-11-03T19:29:32.682Z] 19:29:32     INFO -  package>     26 e4 13 4a 60 75 10 1a 49 34
[task 2018-11-03T19:29:32.683Z] 19:29:32     INFO -  package>     64 54 f5 5e 7d cf f0 f9 c6 fa
[task 2018-11-03T19:29:32.683Z] 19:29:32     INFO -  package>     f8 d5 4f f1 37 1c ed 5a e0 47
[task 2018-11-03T19:29:32.683Z] 19:29:32     INFO -  package>     28 e9
[task 2018-11-03T19:29:32.683Z] 19:29:32     INFO -  package>   signature: 64 bytes
[task 2018-11-03T19:29:32.684Z] 19:29:32     INFO -  package>     7e 9f 10 16 af 82 61 11 37 2f
[task 2018-11-03T19:29:32.684Z] 19:29:32     INFO -  package>     3d b6 29 2c 27 43 b2 51 3c ac
[task 2018-11-03T19:29:32.684Z] 19:29:32     INFO -  package>     c7 b2 48 32 0a ae 9a 95 01 40
[task 2018-11-03T19:29:32.684Z] 19:29:32     INFO -  package>     9b b7 51 ff 3f a0 30 40 f8 8c
[task 2018-11-03T19:29:32.685Z] 19:29:32     INFO -  package>     c9 92 2c 26 8b e1 21 0c 15 79
[task 2018-11-03T19:29:32.685Z] 19:29:32     INFO -  package>     4b d2 34 2b 86 40 f1 83 90 1a
[task 2018-11-03T19:29:32.685Z] 19:29:32     INFO -  package>     39 67 1f d6
[task 2018-11-03T19:29:32.686Z] 19:29:32     INFO -  package> moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB

Looks like it has started failing with this push: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&searchStr=bocf&selectedJob=209661484&revision=f9f879bd247c8493c5f2236c1ddff739dc2f7c1b with Bug 1376873. 
@dminor, is there any chance you could take a look and see if any of your changes caused this? Thank you.
Flags: needinfo?(dminor)
The backout of bug 1376873 didn't fix this.

Earlier in the log:

[task 2018-11-03T10:31:05.121Z] 10:31:05     INFO -  package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox'
[task 2018-11-03T10:31:05.121Z] 10:31:05     INFO -  package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive  gtest '/builds/worker/artifacts/target.gtest.tests.zip'
[task 2018-11-03T10:31:05.121Z] 10:31:05     INFO -  package-tests> Traceback (most recent call last):
[task 2018-11-03T10:31:05.121Z] 10:31:05     INFO -  package-tests>   File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     "__main__", fname, loader, pkg_name)
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>   File "/usr/lib/python2.7/runpy.py", line 72, in _run_code
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     exec code in run_globals
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>   File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/test_archive.py", line 782, in <module>
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     sys.exit(main(sys.argv[1:]))
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>   File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/test_archive.py", line 769, in main
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     skip_duplicates=True)
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>   File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/mozjar.py", line 626, in add
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     deflater.write(data)
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>   File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/mozjar.py", line 717, in write
[task 2018-11-03T10:31:05.122Z] 10:31:05     INFO -  package-tests>     self._deflated.write(self._deflater.compress(data))
[task 2018-11-03T10:31:05.123Z] 10:31:05     INFO -  package-tests> OverflowError: size does not fit in an int

Before the failure, the target.gtest.tests.zip is 696 MB: https://queue.taskcluster.net/v1/task/Bu-mUF4RQTaH6SoI3KDttg/runs/0/artifacts/public/build/target.gtest.tests.zip
Clearing ni since it seems like this is unrelated to Bug 1376873.
Flags: needinfo?(dminor)
Randell, has the gtest package size for fuzzing builds jumped here for unknown reasons?
Flags: needinfo?(rjesup)
Not that I know of... but I really wouldn't know.  I can believe that bug 1376873 would bump the size...
Flags: needinfo?(rjesup)
Mike, this permafails with what looks like a too big file size for gtests. It started when bug 1376873 landed but kept failing when it got backed out. Any ideas? Thank you in advance.
Flags: needinfo?(mh+mozilla)
Summary: Perma-failing make[1]: *** [automation/package-tests] Error 2 → Perma-failing Bocf make[1]: *** [automation/package-tests] Error 2
Setting this as blocking bug 1376873 as we should figure out what is going on here before I reland.
Blocks: 1376873
> OverflowError: size does not fit in an int
> 
> Before the failure, the target.gtest.tests.zip is 696 MB:
> https://queue.taskcluster.net/v1/task/Bu-mUF4RQTaH6SoI3KDttg/runs/0/
> artifacts/public/build/target.gtest.tests.zip

The libxul.so in that zip file is 1.9996 GB. At this point nearly any code change would put it over the limit.
See Also: → 1484073
We could just apply the same workaround as bug 1484073.
I don't know if the workaround from bug 1484073 would be adequate for those fuzzing builds. Maybe they do get some added value from the debug info. However, instead of --enable-debug-symbols=-g1, we could add something like export LDFLAGS=-Wl,--compress-debug-sections=zlib, which will make the linker compress the debug info sections, hopefully making the library much smaller.
Flags: needinfo?(mh+mozilla)
Or produce tarballs instead of zip files. Our code for creating (deterministic) tarballs handles large files correctly. https://hg.mozilla.org/mozilla-central/file/d87537833fcf/python/mozbuild/mozpack/archive.py

IMO the only justifiable reason for using zip files over tarballs is when consumers are accessing a subset of the files in the archive and/or accessing files is non-linear order. In cases where every file in the archive is extracted, tarballs are a superior solution because they will yield better compression ratios and give us the flexibility to use faster compression formats, namely zstandard.
See Also: → 1507094
(In reply to Mike Hommey [:glandium] from comment #10)
> I don't know if the workaround from bug 1484073 would be adequate for those
> fuzzing builds. Maybe they do get some added value from the debug info.
> However, instead of --enable-debug-symbols=-g1, we could add something like
> export LDFLAGS=-Wl,--compress-debug-sections=zlib, which will make the
> linker compress the debug info sections, hopefully making the library much
> smaller.

I've tested with this flag locally and ASan still works with this, so I don't think there is a reason to not try this as an intermediate solution until we can use tarballs instead of zip files (which seems like the right solution to me).

In my test, I used a smaller binary and got around 4.35% size reduction using that linker flag, while ASan still worked afterwards. Note that the major size blow up here doesn't come from debug information (ASan only uses -gline-tables-only) but from the coverage instrumentation itself.
Attached patch Patch (obsolete) — — Splinter Review 

        Attachment #9025103 -
        Flags: review?(choller)

    
    

    
  
It looks like it fixes the asan fuzzing ccov build, but the non-asan fuzzing ccov build is broken:
> [task 2018-11-14T18:01:41.217Z] 18:01:41  WARNING - ../toolkit/crashreporter/tools/unit-symbolstore.py::TestFunctional::testSymbolstore TEST-UNEXPECTED-FAIL
...
> [task 2018-11-14T18:01:41.222Z] 18:01:41     INFO - E   AssertionError: should have a FILE line for nsBrowserApp.cpp

        Attachment #9025103 -
        Flags: review?(choller) → review+

        Attachment #9025299 -
        Attachment is patch: true

        Attachment #9025299 -
        Attachment mime type: application/octet-stream → text/plain

        Attachment #9025299 -
        Flags: review?(choller) → review+

    
    

    
  
Pushed by mcastelluccio@mozilla.com:
https://hg.mozilla.org/integration/mozilla-inbound/rev/70adf7cb715d
Compress debug info section to avoid package failure due to too big libxul.so. r=decoder
https://hg.mozilla.org/integration/mozilla-inbound/rev/7c3b738cd87c
Make fuzzing ccov mozconfig import the code coverage opt mozconfig. r=decoder

    
    

    
  
(In reply to Marco Castelluccio [:marco] from comment #16)
> It looks like it fixes the asan fuzzing ccov build, but the non-asan fuzzing
> ccov build is broken:
> > [task 2018-11-14T18:01:41.217Z] 18:01:41  WARNING - ../toolkit/crashreporter/tools/unit-symbolstore.py::TestFunctional::testSymbolstore TEST-UNEXPECTED-FAIL
> ...
> > [task 2018-11-14T18:01:41.222Z] 18:01:41     INFO - E   AssertionError: should have a FILE line for nsBrowserApp.cpp

You could just disable this test for coverage builds. It's testing behavior around the symbol files we use for crash reporting which isn't relevant for them.

    
    

    
  
https://hg.mozilla.org/mozilla-central/rev/70adf7cb715d
https://hg.mozilla.org/mozilla-central/rev/7c3b738cd87c
Status: NEW → RESOLVED
Closed: 6 years ago

          status-firefox65:
          --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla65
Assignee: nobody → mcastelluccio

    
    

    
      
  

          You need to log in
          before you can comment on or make changes to this bug.
        






  

    
  







  

    

      
Attachment

      

      
      
      
      
    

    

      

        

          

            
General

            

              Creator: 



            

            
Created: 

            
Updated: 

            
Size: