Closed
Bug 1484073
Opened 6 years ago
Closed 6 years ago
Perma-failing tier2 make[1]: *** [automation/package-tests] Error 2
Categories
(Firefox Build System :: General, defect, P5)
Firefox Build System
General
Tracking
(firefox64 fixed)
RESOLVED
FIXED
mozilla64
Tracking | Status | |
---|---|---|
firefox64 | --- | fixed |
People
(Reporter: intermittent-bug-filer, Assigned: marco)
References
(Blocks 1 open bug)
Details
(Keywords: in-triage, intermittent-failure, Whiteboard: [stockwell unknown])
Attachments
(1 file)
1.08 KB,
patch
|
glandium
:
review+
|
Details | Diff | Splinter Review |
Filed by: apavel [at] mozilla.com https://treeherder.mozilla.org/logviewer.html#?job_id=194267582&repo=mozilla-central https://queue.taskcluster.net/v1/task/bTFhecMUQqyNdFDpA5KyoA/runs/0/artifacts/public/logs/live_backing.log [task 2018-08-16T10:31:56.084Z] 10:31:56 INFO - package-tests> make[2]: *** [package-tests-gtest] Error 1 [task 2018-08-16T10:31:56.084Z] 10:31:56 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:31:56.084Z] 10:31:56 INFO - package-tests> make[2]: *** Waiting for unfinished jobs.... [task 2018-08-16T10:31:56.280Z] 10:31:56 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:31:56.281Z] 10:31:56 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive raptor '/builds/worker/artifacts/target.raptor.tests.tar.gz' [task 2018-08-16T10:31:56.281Z] 10:31:56 INFO - package-tests> Wrote 924 files in 6987259 bytes to target.raptor.tests.tar.gz in 5.52s [task 2018-08-16T10:31:56.281Z] 10:31:56 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:31:56.778Z] 10:31:56 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:31:56.778Z] 10:31:56 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive talos '/builds/worker/artifacts/target.talos.tests.tar.gz' [task 2018-08-16T10:31:56.778Z] 10:31:56 INFO - package-tests> Wrote 1371 files in 15609520 bytes to target.talos.tests.tar.gz in 6.02s [task 2018-08-16T10:31:56.778Z] 10:31:56 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:06.713Z] 10:32:06 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:06.713Z] 10:32:06 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive cppunittest '/builds/worker/artifacts/target.cppunittest.tests.tar.gz' [task 2018-08-16T10:32:06.713Z] 10:32:06 INFO - package-tests> Wrote 64 files in 142439118 bytes to target.cppunittest.tests.tar.gz in 15.99s [task 2018-08-16T10:32:06.714Z] 10:32:06 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:10.486Z] 10:32:10 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:10.486Z] 10:32:10 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive xpcshell '/builds/worker/artifacts/target.xpcshell.tests.tar.gz' [task 2018-08-16T10:32:10.487Z] 10:32:10 INFO - package-tests> Wrote 5868 files in 11000486 bytes to target.xpcshell.tests.tar.gz in 19.73s [task 2018-08-16T10:32:10.487Z] 10:32:10 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:59.579Z] 10:32:59 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:32:59.579Z] 10:32:59 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive common '/builds/worker/artifacts/target.common.tests.tar.gz' [task 2018-08-16T10:32:59.579Z] 10:32:59 INFO - package-tests> Wrote 45023 files in 40610305 bytes to target.common.tests.tar.gz in 68.79s [task 2018-08-16T10:32:59.579Z] 10:32:59 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:17.582Z] 10:33:17 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:17.582Z] 10:33:17 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive reftest '/builds/worker/artifacts/target.reftest.tests.tar.gz' [task 2018-08-16T10:33:17.582Z] 10:33:17 INFO - package-tests> Wrote 30477 files in 47757980 bytes to target.reftest.tests.tar.gz in 86.83s [task 2018-08-16T10:33:17.582Z] 10:33:17 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:17.630Z] 10:33:17 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:17.630Z] 10:33:17 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive mochitest '/builds/worker/artifacts/target.mochitest.tests.tar.gz' [task 2018-08-16T10:33:17.630Z] 10:33:17 INFO - package-tests> Wrote 30165 files in 58348578 bytes to target.mochitest.tests.tar.gz in 86.86s [task 2018-08-16T10:33:17.630Z] 10:33:17 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive web-platform '/builds/worker/artifacts/target.web-platform.tests.tar.gz' [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - package-tests> Wrote 66998 files in 53753804 bytes to target.web-platform.tests.tar.gz in 111.15s [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - package-tests> make[2]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox' [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - /builds/worker/workspace/build/src/build/moz-automation.mk:89: recipe for target 'automation/package-tests' failed [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - make[1]: *** [automation/package-tests] Error 2 [task 2018-08-16T10:33:41.945Z] 10:33:41 INFO - make[1]: *** Waiting for unfinished jobs.... [task 2018-08-16T10:35:40.657Z] 10:35:40 INFO - package> make[5]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/browser/installer' [task 2018-08-16T10:35:40.657Z] 10:35:40 INFO - package> OMNIJAR_NAME=omni.ja \ [task 2018-08-16T10:35:40.658Z] 10:35:40 INFO - package> NO_PKG_FILES="core bsdecho js js-config jscpucfg nsinstall viewer TestGtkEmbed elf-dynstr-gc mangle* maptsv* mfc* msdump* msmap* nm2tsv* nsinstall* res/samples res/throbber shlibsign* certutil* pk12util* BadCertServer* OCSPStaplingServer* SymantecSanctionsServer* GenerateOCSPResponse* chrome/chrome.rdf chrome/app-chrome.manifest chrome/overlayinfo components/compreg.dat components/xpti.dat content_unit_tests necko_unit_tests *.dSYM " \ [task 2018-08-16T10:35:40.665Z] 10:35:40 INFO - package> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python /builds/worker/workspace/build/src/toolkit/mozapps/installer/packager.py -DPKG_LOCALE_MANIFEST=/builds/worker/workspace/build/src/obj-firefox/browser/installer/locale-manifest.in -DMOZ_APP_NAME=firefox -DPREF_DIR=defaults/preferences -DMOZ_DEBUG=1 -DMOZ_GTK=1 -DMOZ_GTK3=1 -DJAREXT= -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_CHILD_PROCESS_NAME=plugin-container -DNECKO_WIFI -DDLL_PREFIX=lib -DDLL_SUFFIX=.so -DBIN_SUFFIX= -DDIR_MACOS= -DDIR_RESOURCES= -DBINPATH='bin' -DRESPATH='bin' -DLPROJ_ROOT=en -DENABLE_MARIONETTE=1 -DA11Y_LOG=1 -DACCESSIBILITY=1 -DATK_MAJOR_VERSION=2 -DATK_MINOR_VERSION=4 -DATK_REV_VERSION=0 -DBROWSER_CHROME_URL=chrome://browser/content/browser.xul -DBROWSER_CHROME_URL_QUOTED='"chrome://browser/content/browser.xul"' -DBUILD_CTYPES=1 -DCROSS_COMPILE='' -DEARLY_BETA_OR_EARLIER=1 -DENABLE_INTL_API=1 -DENABLE_SYSTEM_EXTENSION_DIRS=1 -DENABLE_TESTS=1 -DEXPOSE_INTL_API=1 -DFIREFOX_VERSION=63.0a1 -DFORCE_PR_LOG=1 -DFUNCPROTO=15 -DGDK_VERSION_MAX_ALLOWED=GDK_VERSION_3_4 -DGLIB_VERSION_MAX_ALLOWED=GLIB_VERSION_2_32 -DGLIB_VERSION_MIN_REQUIRED=GLIB_VERSION_2_26 -DGL_PROVIDER_GLX=1 -DGTEST_HAS_RTTI=0 -DHAVE_64BIT_BUILD=1 -DHAVE_ALLOCA_H=1 -DHAVE_BYTESWAP_H=1 -DHAVE_CLOCK_MONOTONIC=1 -DHAVE_CPUID_H=1 -DHAVE_DIRENT_H=1 -DHAVE_DLADDR=1 -DHAVE_DLOPEN=1 -DHAVE_FONTCONFIG_FCFREETYPE_H=1 -DHAVE_FT_BITMAP_SIZE_Y_PPEM=1 -DHAVE_FT_GLYPHSLOT_EMBOLDEN=1 -DHAVE_FT_LOAD_SFNT_TABLE=1 -DHAVE_GETOPT_H=1 -DHAVE_GMTIME_R=1 -DHAVE_INTTYPES_H=1 -DHAVE_LCHOWN=1 -DHAVE_LINUX_IF_ADDR_H=1 -DHAVE_LINUX_PERF_EVENT_H=1 -DHAVE_LINUX_QUOTA_H=1 -DHAVE_LINUX_RTNETLINK_H=1 -DHAVE_LOCALECONV=1 -DHAVE_LOCALTIME_R=1 -DHAVE_LSTAT64=1 -DHAVE_MALLINFO=1 -DHAVE_MALLOC_H=1 -DHAVE_MALLOC_USABLE_SIZE=1 -DHAVE_MEMALIGN=1 -DHAVE_MEMMEM=1 -DHAVE_NETINET_IN_H=1 -DHAVE_NL_TYPES_H=1 -DHAVE_POSIX_FADVISE=1 -DHAVE_POSIX_FALLOCATE=1 -DHAVE_POSIX_MEMALIGN=1 -DHAVE_PTHREAD_H=1 -DHAVE_RES_NINIT=1 -DHAVE_SETPRIORITY=1 -DHAVE_STAT64=1 -DHAVE_STDINT_H=1 -DHAVE_STRERROR=1 -DHAVE_STRNDUP=1 -DHAVE_SYSCALL=1 -DHAVE_SYS_QUEUE_H=1 -DHAVE_SYS_QUOTA_H=1 -DHAVE_SYS_TYPES_H=1 -DHAVE_THREAD_TLS_KEYWORD=1 -DHAVE_TRUNCATE64=1 -DHAVE_UNISTD_H=1 -DHAVE_VALLOC=1 -DHAVE_VA_COPY=1 -DHAVE_VA_LIST_AS_ARRAY=1 -DHAVE_VISIBILITY_ATTRIBUTE=1 -DHAVE_VISIBILITY_HIDDEN_ATTRIBUTE=1 -DHAVE__UNWIND_BACKTRACE=1 -DHAVE___CXA_DEMANGLE=1 -DJS_64BIT=1 -DJS_BUILD_BINAST=1 -DJS_CODEGEN_X64=1 -DJS_DEFAULT_JITREPORT_GRANULARITY=3 -DJS_GC_ZEAL=1 -DJS_JITSPEW=1 -DJS_PUNBOX64=1 -DJS_TRACE_LOGGING=1 -DMALLOC_H='<malloc.h>' -DMALLOC_USABLE_SIZE_CONST_PTR='' -DMOZILLA_OFFICIAL=1 -DMOZILLA_UAVERSION='"63.0"' -DMOZILLA_VERSION='"63.0a1"' -DMOZILLA_VERSION_U=63.0a1 -DMOZ_ACCESSIBILITY_ATK=1 -DMOZ_ALLOW_LEGACY_EXTENSIONS=1 -DMOZ_APP_UA_NAME='""' -DMOZ_APP_UA_VERSION='"63.0a1"' -DMOZ_AV1=1 -DMOZ_BUILD_APP=browser -DMOZ_BUILD_WEBRENDER=1 -DMOZ_BUNDLED_FONTS=1 -DMOZ_CODE_COVERAGE=1 -DMOZ_CRASHREPORTER=1 -DMOZ_DATA_REPORTING=1 -DMOZ_DEBUG=1 -DMOZ_DEBUG_RUST=1 -DMOZ_DEMANGLE_SYMBOLS=1 -DMOZ_DISTRIBUTION_ID='"org.mozilla"' -DMOZ_DLL_SUFFIX='".so"' -DMOZ_DUMP_PAINTING=1 -DMOZ_ENABLE_DBUS=1 -DMOZ_ENABLE_GCONF=1 -DMOZ_ENABLE_SIGNMAR=1 -DMOZ_ENABLE_SKIA=1 -DMOZ_ENABLE_SKIA_PDF=1 -DMOZ_ENABLE_SKIA_PDF_SFNTLY=1 -DMOZ_ENABLE_XREMOTE=1 -DMOZ_FEEDS=1 -DMOZ_FFMPEG=1 -DMOZ_FFVPX=1 -DMOZ_FMP4=1 -DMOZ_GECKO_PROFILER=1 -DMOZ_GLUE_IN_PROGRAM=1 -DMOZ_INSTRUMENT_EVENT_LOOP=1 -DMOZ_LIBAV_FFT=1 -DMOZ_LOGGING=1 -DMOZ_MACBUNDLE_ID=org.mozilla.nightlydebug -DMOZ_MEMORY=1 -DMOZ_PEERCONNECTION=1 -DMOZ_PHOENIX=1 -DMOZ_PLACES=1 -DMOZ_PULSEAUDIO=1 -DMOZ_RAW=1 -DMOZ_REFLOW_PERF=1 -DMOZ_REFLOW_PERF_DSP=1 -DMOZ_REPLACE_MALLOC=1 -DMOZ_RUST_SIMD=1 -DMOZ_SAMPLE_TYPE_FLOAT32=1 -DMOZ_SCTP=1 -DMOZ_SERVICES_HEALTHREPORT=1 -DMOZ_SRTP=1 -DMOZ_STATIC_JS=1 -DMOZ_TELEMETRY_ON_BY_DEFAULT=1 -DMOZ_TELEMETRY_REPORTING=1 -DMOZ_TREE_CAIRO=1 -DMOZ_TREE_PIXMAN=1 -DMOZ_UPDATER=1 -DMOZ_UPDATE_CHANNEL=default -DMOZ_USER_DIR='".mozilla"' -DMOZ_VERIFY_MAR_SIGNATURE=1 -DMOZ_VORBIS=1 -DMOZ_VPX_NO_MEM_REPORTING=1 -DMOZ_WEBM_ENCODER=1 -DMOZ_WEBRTC=1 -DMOZ_WEBRTC_ASSERT_ALWAYS=1 -DMOZ_WEBRTC_SIGNALING=1 -DMOZ_WEBSPEECH=1 -DMOZ_WEBSPEECH_TEST_BACKEND=1 -DMOZ_WIDGET_GTK=3 -DMOZ_X11=1 -DMOZ_XUL=1 -DNIGHTLY_BUILD=1 -DNO_NSPR_10_SUPPORT=1 -DNS_PRINTING=1 -DNS_PRINT_PREVIEW=1 -DSTATIC_JS_API=1 -DSTDC_HEADERS=1 -DTARGET_XPCOM_ABI='"x86_64-gcc3"' -DUSE_SKIA=1 -DUSE_SKIA_GPU=1 -DU_STATIC_IMPLEMENTATION=1 -DU_USING_ICU_NAMESPACE=0 -DVA_COPY=va_copy -DVPX_X86_ASM=1 -DXP_LINUX=1 -DXP_UNIX=1 -D_REENTRANT=1 -DAB_CD=en-US \ [task 2018-08-16T10:35:40.665Z] 10:35:40 INFO - package> --format omni \ [task 2018-08-16T10:35:40.665Z] 10:35:40 INFO - package> --removals /builds/worker/workspace/build/src/browser/installer/removed-files.in \ [task 2018-08-16T10:35:40.665Z] 10:35:40 INFO - package> \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> --minify \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> --optimizejars \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> --compress none \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> /builds/worker/workspace/build/src/browser/installer/package-manifest.in '../../dist' '../../dist'/firefox \ [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> Executing /builds/worker/workspace/build/src/obj-firefox/dist/bin/shlibsign -v -o ../../dist/firefox/libsoftokn3.chk -i ../../dist/firefox/libsoftokn3.so [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> Library File: ../../dist/firefox/libsoftokn3.so 1473936 bytes [task 2018-08-16T10:35:40.666Z] 10:35:40 INFO - package> Check File: ../../dist/firefox/libsoftokn3.chk [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> hash: 32 bytes [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> ee cf 8f c5 2f 78 e8 91 01 c8 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> 20 1c 81 b0 d8 dd 4b d2 59 06 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> 73 06 70 d9 cf ef af fc 56 2f [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> da c8 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> signature: 64 bytes [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> 74 b9 a7 a4 1b 20 16 24 64 e7 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> ca 11 b4 ff c5 aa 13 9e 26 07 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> ae 57 2e 63 5d a4 da 7f d2 30 [task 2018-08-16T10:35:40.667Z] 10:35:40 INFO - package> fa 92 22 26 37 74 2f aa 33 51 [task 2018-08-16T10:35:40.668Z] 10:35:40 INFO - package> 68 ab 3a 52 cb 90 60 4b df 65 [task 2018-08-16T10:35:40.668Z] 10:35:40 INFO - package> 63 dd 0c 19 01 b2 4e 93 db 08 [task 2018-08-16T10:35:40.668Z] 10:35:40 INFO - package> 5d c6 75 0b [task 2018-08-16T10:35:40.668Z] 10:35:40 INFO - package> moduleSpec configdir='' certPrefix='' keyPrefix='' secmod='' flags=noCertDB, noModDB Karl can this be related to Bug 1470914?
Updated•6 years ago
|
Summary: Intermittent make[1]: *** [automation/package-tests] Error 2 → Perma-failing tier2 make[1]: *** [automation/package-tests] Error 2
Comment 2•6 years ago
|
||
On first glance I don't see a connection to NSS. The first indication of failure seems to be above the quoted lines. From the raw log: INFO - package-tests> make[2]: Entering directory '/builds/worker/workspace/build/src/obj-firefox' INFO - package-tests> /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.test_archive gtest '/builds/worker/artifacts/target.gtest.tests.zip' INFO - package-tests> Traceback (most recent call last): INFO - package-tests> File "/usr/lib/python2.7/runpy.py", line 162, in _run_module_as_main INFO - package-tests> "__main__", fname, loader, pkg_name) INFO - package-tests> File "/usr/lib/python2.7/runpy.py", line 72, in _run_code INFO - package-tests> exec code in run_globals INFO - package-tests> File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/test_archive.py", line 734, in <module> INFO - package-tests> sys.exit(main(sys.argv[1:])) INFO - package-tests> File "/builds/worker/workspace/build/src/python/mozbuild/mozbuild/action/test_archive.py", line 721, in main INFO - package-tests> skip_duplicates=True) INFO - package-tests> File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/mozjar.py", line 626, in add INFO - package-tests> deflater.write(data) INFO - package-tests> File "/builds/worker/workspace/build/src/python/mozbuild/mozpack/mozjar.py", line 717, in write INFO - package-tests> self._deflated.write(self._deflater.compress(data)) INFO - package-tests> OverflowError: size does not fit in an int INFO - package-tests> /builds/worker/workspace/build/src/testing/testsuite-targets.mk:164: recipe for target 'package-tests-gtest' failed INFO - package-tests> make[2]: *** [package-tests-gtest] Error 1 Apparently the cause is: OverflowError: size does not fit in an int Lines 717 and 626 were last touched by glandium in 2013. Maybe a recent size increase causes an overflow? I suggest to ask someone with experience on the mozbuild/mozpack python code to investigate what's going on.
Flags: needinfo?(kaie)
Comment 3•6 years ago
|
||
(In reply to Kai Engert (:kaie:) from comment #2) > Apparently the cause is: > OverflowError: size does not fit in an int > > Lines 717 and 626 were last touched by glandium in 2013. FYI, bug 780561. https://hg.mozilla.org/mozilla-central/rev/1bbce6c7e006
Comment 4•6 years ago
|
||
The size of libxul.so in target.gtest.tests.zip was 2146269280. That is exactly 1214367 less than 2^31-1, which is the maximum size the zip writer can store. So, somewhere between the previous merge and that merge made the size of libxul tip over the limit. The file contains all debug info, and is 293M once stripped. Do we really need to have the full debug info on this file?
Flags: needinfo?(rforbes)
Flags: needinfo?(mcastelluccio)
Comment 5•6 years ago
|
||
The actual limit of the zip writer code is 2^32 - 1, but something else must be bailing out before that... maybe zlib. Part of the problem being that the code doesn't try to stream anything. For a quick fix, it's probably easier to figure out how to reduce the size of this file.
Comment 6•6 years ago
|
||
It's worth noting that it's green now. Something might have made libxul.so slightly smaller in the meanwhile. But it's bound to happen again some time.
Comment hidden (Intermittent Failures Robot) |
Comment 8•6 years ago
|
||
This seems to be failing consistently again: https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&filter-searchStr=linux%2064%20debug%20ccov Here's the first push on central where it started failing: https://hg.mozilla.org/mozilla-central/pushloghtml?changeset=a75079e6f2345d2cad5623b3973a731c431dec19
Comment 9•6 years ago
|
||
I am going to leave this for :marco to answer. This is actually his build.
Flags: needinfo?(rforbes)
Comment 10•6 years ago
|
||
It looks like one the patches in this push fixed the problem (maybe the RUSTFLAGS patch?): https://treeherder.mozilla.org/#/jobs?repo=mozilla-central&revision=9f3d83072326cc96b3c26c5edd67ea277d6fa86c&filter-searchStr=ccov
Comment 11•6 years ago
|
||
I think glandium's question in comment 4 stands--even if this is temporarily fixed, if we don't need the full unstripped gtest libxul.so we shouldn't be packaging it. We can almost certainly get away with partially stripping it without breaking anything, and that will not only help us avoid this in the future but also save time and storage cost shuffling around an enormous binary.
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 13•6 years ago
|
||
I guess we don't need the full debug info, how do we partially strip it?
Flags: needinfo?(mcastelluccio) → needinfo?(ted)
Comment hidden (Intermittent Failures Robot) |
Comment 15•6 years ago
|
||
In the last 7 days there have been 35 failures, on Linux 64 ccov, mostly debug and some opt.
Updated•6 years ago
|
Whiteboard: [stockwell needswork]
Comment hidden (Intermittent Failures Robot) |
Comment 17•6 years ago
|
||
I did some retriggers and seems that the fail started from Bug 1420996 - https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=success,running,pending,testfailed,busted,exception,runnable&classifiedState=unclassified&searchStr=linux,x64,ccov,debug,build-linux64-ccov%2Fdebug&revision=712cd59c243762c23248cc07c6fcf300fa2bf65a Retriggs: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&resultStatus=success,running,pending,testfailed,busted,exception,runnable&classifiedState=unclassified&searchStr=linux,x64,ccov,debug,build-linux64-ccov%2Fdebug&tochange=4b793141b127945f9e69fbbe92cca4bf2850db64&fromchange=582f4fc155f7d7549171cfc22a3ca1bb063acca2 Botond, can you please take a look?
Flags: needinfo?(botond)
Comment hidden (Intermittent Failures Robot) |
Comment 19•6 years ago
|
||
I recommend backing out the patches from bug 1420996, they have permanently broken linux64 ccov builds since September 26th and we are not getting data for this platform. Hopefully Botond has an idea of what is going on- based on this history of this bug, it is a tooling issue and we might just have crossed a line by adding more code.
Assignee | ||
Comment 20•6 years ago
|
||
Bug 1494671 is likely related.
Comment 21•6 years ago
|
||
Based on the previous comments in this bug, it seems like the issue is the size of libxul.so, which is close to some limit imposed by tooling, crossing that limit. If bug 1420996 has increased the size of libxul.so, that may be addressed by bug 1494671. However, I don't think either bug 1494671 or backing out bug 1420996 would be a proper solution to this issue, only a temporary one. The size of libxul.so is expected to increase over time as functionality is added to Gecko, so soon enough another bug will get us over the limit and we'll be back to the same problem.
Flags: needinfo?(botond)
Comment hidden (Intermittent Failures Robot) |
Assignee | ||
Comment 23•6 years ago
|
||
Assignee: nobody → mcastelluccio
Status: NEW → ASSIGNED
Flags: needinfo?(ted)
Attachment #9013569 -
Flags: review?(mh+mozilla)
Updated•6 years ago
|
Attachment #9013569 -
Flags: review?(mh+mozilla) → review+
Comment 24•6 years ago
|
||
Pushed by mcastelluccio@mozilla.com: https://hg.mozilla.org/integration/mozilla-inbound/rev/5f6fd031d3a6 Emit less detailed debug info for Linux coverage builds. r=glandium
Comment 25•6 years ago
|
||
bugherder |
https://hg.mozilla.org/mozilla-central/rev/5f6fd031d3a6
Status: ASSIGNED → RESOLVED
Closed: 6 years ago
status-firefox64:
--- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla64
Comment 26•6 years ago
|
||
FWIW, if you wanted to actually change how we strip the gtest libxul, what's happening now is that you have `--disable-install-strip` in your ccov mozconfigs: https://hg.mozilla.org/integration/mozilla-inbound/file/5f6fd031d3a6/browser/config/mozconfigs/linux64/code-coverage#l7 and that sets `PKG_SKIP_STRIP=1` in the build system, which triggers the conditional here that just copies the gtest libxul: https://dxr.mozilla.org/mozilla-central/rev/56b988a937689d5599400afa59b72c390b40abf2/testing/testsuite-targets.mk#201-220 An alternative would be to stop doing that, and instead set `STRIP_FLAGS` to something that preserves the info you need. For builds with `--enable-profiling` configure sets `STRIP_FLAGS=--strip-debug` by default: https://dxr.mozilla.org/mozilla-central/rev/56b988a937689d5599400afa59b72c390b40abf2/old-configure.in#654
Comment hidden (Intermittent Failures Robot) |
Comment hidden (Intermittent Failures Robot) |
You need to log in
before you can comment on or make changes to this bug.
Description
•