Closed Bug 1536194 Opened 5 years ago Closed 5 years ago

High Frequency OS X cross compiled addon [taskcluster:error] Task timeout after 3600 seconds. Force killing container.

Categories

(Firefox Build System :: General, defect, P5)

Product:
Firefox Build System
Component:
General
Type:
defect

Tracking

(firefox-esr60 unaffected, firefox66 unaffected, firefox67 fixed, firefox68 fixed)

Status:
RESOLVED FIXED
Milestone:
mozilla68
Tracking Flags:
Tracking Status
firefox-esr60 --- unaffected
firefox66 --- unaffected
firefox67 --- fixed
firefox68 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: mshal)

References

(Regression)

Details

(Keywords: in-triage, intermittent-failure, regression)

Attachments

(1 file)

Bug 1536194 - Fix uses of MOZ_LTO in a few mozconfigs; r?#firefox-build-system-reviewers
47 bytes, text/x-phabricator-request
pascalc
: approval-mozilla-beta+
Details | Review

#[markdown(off)]
Filed by: ncsoregi [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=234564250&repo=mozilla-beta

https://queue.taskcluster.net/v1/task/bi4gsttKRTeMnA5RMjBHMg/runs/0/artifacts/public/logs/live_backing.log

[task 2019-03-18T16:44:08.226Z] 16:44:08 INFO - Starting Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:08.226Z] 16:44:08 INFO - Running Mac pre-processing on file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:08.226Z] 16:44:08 INFO - /builds/worker/workspace/build/src/build/macosx/llvm-dsymutil --arch=x86_64 /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:08.227Z] 16:44:08 INFO - warning: could not find referenced DIE
[task 2019-03-18T16:44:08.227Z] 16:44:08 INFO - note: while processing /builds/worker/workspace/build/src/obj-firefox/x86_64-apple-darwin/release/libgkrust.a(gkrust-9e7e3fc0c5ddbbbd.gkrust.lqkbllc7-cgu.0.rcgu.o)
...
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - note: while processing /builds/worker/workspace/build/src/obj-firefox/x86_64-apple-darwin/release/libgkrust.a(gkrust-9e7e3fc0c5ddbbbd.gkrust.lqkbllc7-cgu.0.rcgu.o)
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - warning: inconsistent range data.
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - note: while processing emitting debug_ranges
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - warning: inconsistent range data.
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - note: while processing emitting debug_ranges
[task 2019-03-18T16:44:22.281Z] 16:44:22 INFO - warning: inconsistent range data.
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - note: while processing emitting debug_ranges
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - warning: inconsistent range data.
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - note: while processing emitting debug_ranges
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - Finished processing /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL in 143.62s
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - Beginning work for file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:22.282Z] 16:44:22 INFO - Processing file: /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:22.283Z] 16:44:22 INFO - /builds/worker/workspace/build/src/obj-firefox/dist/host/bin/dump_syms -a x86_64 -g /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL.dSYM /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL
[task 2019-03-18T16:44:22.283Z] 16:44:22 INFO - PERFHERDER_DATA: {"framework": {"name": "build_metrics"}, "suites": [{"subtests": [{"alertChangeType": "absolute", "name": "num_static_constructors", "value": 84, "alertThreshold": 3}], "name": "compiler_metrics"}]}
[task 2019-03-18T16:44:22.283Z] 16:44:22 INFO - Finished processing /builds/worker/workspace/build/src/obj-firefox/toolkit/library/XUL in 327.94s
[task 2019-03-18T16:44:22.283Z] 16:44:22 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library'
[task 2019-03-18T16:50:47.864Z] 16:50:47 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:47.865Z] 16:50:47 INFO - /builds/worker/workspace/build/src/clang/bin/clang++ -isysroot /builds/worker/workspace/build/src/MacOSX10.11.sdk --target=x86_64-darwin11 -Qunused-arguments -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -Qunused-arguments -Wall -Wbitfield-enum-conversion -Wempty-body -Wignored-qualifiers -Woverloaded-virtual -Wpointer-arith -Wshadow-field-in-constructor-modified -Wsign-compare -Wtype-limits -Wunreachable-code -Wunreachable-code-return -Wwrite-strings -Wno-invalid-offsetof -Wclass-varargs -Wfloat-overflow-conversion -Wfloat-zero-conversion -Wloop-analysis -Wc++1z-compat -Wc++2a-compat -Wcomma -Wimplicit-fallthrough -Werror=non-literal-null-conversion -Wstring-conversion -Wtautological-overlap-compare -Wtautological-unsigned-enum-zero-compare -Wtautological-unsigned-zero-compare -Wno-inline-new-delete -Wno-error=deprecated-declarations -Wno-error=array-bounds -Wno-error=return-std-move -Wno-error=atomic-alignment -Wformat -Wformat-security -Wno-gnu-zero-variadic-macro-arguments -Wno-unknown-warning-option -Wno-return-type-c-linkage -fno-sized-deallocation -fcrash-diagnostics-dir=/builds/worker/artifacts -U_FORTIFY_SOURCE -D_FORTIFY_SOURCE=2 -fstack-protector-strong -fno-exceptions -fno-strict-aliasing -stdlib=libc++ -fno-rtti -ffunction-sections -fdata-sections -fno-exceptions -fno-math-errno -pthread -pipe -g -Xclang -load -Xclang /builds/worker/workspace/build/src/obj-firefox/build/clang-plugin/libclang-plugin.so -Xclang -add-plugin -Xclang moz-check -O3 -fno-omit-frame-pointer -funwind-tables -Werror -fPIC -o XUL -Wl,-filelist,/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest/XUL.list -flto=thin -Wl,-syslibroot,/builds/worker/workspace/build/src/MacOSX10.11.sdk -fstack-protector-strong -framework Cocoa -lobjc -framework AudioToolbox -framework ExceptionHandling '-Wl,-U,_OBJC_CLASS_$_NSTouchBar' '-Wl,-U,_OBJC_CLASS_$_NSSharingServicePickerTouchBarItem' '-Wl,-U,_OBJC_METACLASS_$_NSTouchBar' '-Wl,-U,_OBJC_CLASS_$_NSCustomTouchBarItem' -lresolv -Wl,-executable_path,/builds/worker/workspace/build/src/obj-firefox/dist/bin -Wl,-dead_strip -Wl,-object_path_lto,XUL.lto.o/ ../../../security/nss/lib/crmf/crmf_crmf/libcrmf.a ../../../js/src/build/libjs_static.a /builds/worker/workspace/build/src/obj-firefox/x86_64-apple-darwin/release/libgkrust_gtest.a ../../../mozglue/build/libmozglue.dylib ../../../security/libnss3.dylib ../../../config/external/lgpllibs/liblgpllibs.dylib -dynamiclib -install_name @executable_path/XUL -compatibility_version 1 -current_version 1 -single_module -framework IOKit -framework AudioToolbox -framework CoreMedia -framework VideoToolbox -framework Foundation -framework AVFoundation -framework CoreVideo -framework AppKit -framework OpenGL -framework LocalAuthentication -framework Security -lm -framework SystemConfiguration -F/builds/worker/workspace/build/src/MacOSX10.11.sdk/System/Library/PrivateFrameworks -framework CoreUI -framework CoreSymbolication -lcups -Wl,-framework,Foundation -Wl,-framework,CoreFoundation -Wl,-framework,CoreLocation -Wl,-framework,QuartzCore -Wl,-framework,Carbon -Wl,-framework,CoreAudio -Wl,-framework,CoreVideo -Wl,-framework,AudioToolbox -Wl,-framework,AudioUnit -Wl,-framework,AddressBook -Wl,-framework,OpenGL -Wl,-framework,Security -Wl,-framework,ServiceManagement -Wl,-framework,CoreServices -Wl,-framework,ApplicationServices -Wl,-framework,AppKit -framework AudioUnit -framework CoreAudio
[task 2019-03-18T16:50:47.865Z] 16:50:47 INFO - ld: warning: directory not found for option '-L/builds/worker/workspace/src/obj-firefox'
[task 2019-03-18T16:50:47.865Z] 16:50:47 INFO - ld: warning: directory not found for option '-L/builds/worker/workspace/src/gcc/lib64'
[task 2019-03-18T16:50:47.865Z] 16:50:47 INFO - ld: warning: could not create compact unwind for _ffi_call_unix64: does not use RBP or RSP based frame
[task 2019-03-18T16:50:47.866Z] 16:50:47 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - /builds/worker/workspace/build/src/obj-firefox/_virtualenvs/init/bin/python -m mozbuild.action.check_binary --target XUL
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - TEST-PASS | check_nsmodules | XUL
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - chmod +x XUL
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Entering directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - ../../../config/nsinstall -L /builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest -m 644 'XUL' '../../../dist/bin/gtest'
[task 2019-03-18T16:50:52.231Z] 16:50:52 INFO - make[4]: Leaving directory '/builds/worker/workspace/build/src/obj-firefox/toolkit/library/gtest'
[taskcluster:error] Task timeout after 3600 seconds. Force killing container.
[taskcluster 2019-03-18 16:53:47.518Z] === Task Finished ===
[taskcluster 2019-03-18 16:53:47.518Z] Unsuccessful task run with exit code: -1 completed in 3601.635 seconds

Chris, any idea why the build time for that job would have gone up over the last cycle (it only runs on beta)? build times for other OS X jobs on central don't show on upwards trend.

Flags: needinfo?(catlee)

I don't know, sorry. Do we these with sccache on central and not on beta?

Flags: needinfo?(catlee)

sccache is also enabled on beta, but it's better to compare the normal OSX beta builds with the add-on devel ones. Notable differences from looking at one log of each:

  • custom build variant 'add-on-devel' (64_add-on-devel.py)
  • LTO disabled
  • --enable-instruments flag active

Others:

  • full symbols disabled
  • unofficial branding
  • extension signing disabled
  • legacy extensions enabled
  • --disable-install-strip active

Kev, do you remember who set these jobs up and can investigate what's the issue here? Thank you.

Flags: needinfo?(kev)

It would have likely been kmoir, if memory serves.

Flags: needinfo?(kev) → needinfo?(kmoir)
Keywords: in-triage

I'm going to redirect this investigation to mshal.

Assignee: nobody → mshal

Looks like LTO is mis-set on these builds after bug 1530908. Mac ccov builds are also affected, and while looking into this I noticed that I somehow failed to change the linux64 profile-use builds to use the environment variable even though glandium warned me about it :).

I'll post a patch for central, and then we can uplift to beta.

Flags: needinfo?(kmoir)

After bug 1530908 changed LTO to be environment driven, the macosx64
nightly mozconfig enables LTO by setting MOZ_LTO=1. The add-on-devel
mozconfig tried to disable LTO by using --disable-lto, but the
environment variable takes precedence, leaving LTO enabled for these
builds. This pushed the build time up to be close to the max runtime for
the task, causing frequent intermittent failures.

We should 'unset MOZ_LTO' everywhere that --disable-lto was used, and
'export MOZ_LTO=1' everywhere --enable-lto was used.

Pushed by mshal@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/89c293eaf78e
Fix uses of MOZ_LTO in a few mozconfigs; r=firefox-build-system-reviewers,glandium

https://hg.mozilla.org/mozilla-central/rev/89c293eaf78e

Status: NEW → RESOLVED
Closed: 5 years ago
status-firefox68: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla68

Comment on attachment 9056713 [details]
Bug 1536194 - Fix uses of MOZ_LTO in a few mozconfigs; r?#firefox-build-system-reviewers

Beta/Release Uplift Approval Request

  • Feature/Bug causing the regression: Bug 1530908
  • User impact if declined: OSX add-on builds will continue to fail intermittently on beta
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: Yes
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is intended to revert the LTO setting for OSX add-on builds to the previous value (off), which should lower build times below the max runtime threshold.
  • String changes made/needed:
Attachment #9056713 - Flags: approval-mozilla-beta?
Attachment #9056713 - Flags: approval-mozilla-beta? → approval-mozilla-beta+
Has Regression Range: --- → yes
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: