1632680 - Intermittent PROCESS-CRASH | automation.py | application crashed [@ -] | After automation.py | application terminated with exit code 1

Status: RESOLVED WORKSFORME

(Core :: Widget: Cocoa, defect)

Description

[Treeherder Bug Filer]

Filed by: nbeleuzu [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=299106472&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/L8tjbj_ZRiG6pb72KguRLQ/runs/0/artifacts/public/logs/live_backing.log

---

[task 2020-04-23T22:39:05.823Z] 22:39:05 INFO - ###!!! [Child][MessageChannel::SendAndWait] Error: Channel error: cannot send/recv
[task 2020-04-23T22:39:05.823Z] 22:39:05 INFO - ###!!! [Child][MessageChannel] Error: (msgtype=0x63001E,name=PLayerTransaction::Msg_Shutdown) Channel error: cannot send/recv
[task 2020-04-23T22:39:05.824Z] 22:39:05 INFO - Exiting due to channel error.
[task 2020-04-23T22:39:05.824Z] 22:39:05 INFO - Exiting due to channel error.
[task 2020-04-23T22:39:05.824Z] 22:39:05 INFO - Exiting due to channel error.
[task 2020-04-23T22:39:05.824Z] 22:39:05 INFO - Exiting due to channel error.
[task 2020-04-23T22:39:05.825Z] 22:39:05 ERROR - TEST-UNEXPECTED-FAIL | None | application terminated with exit code 1
[task 2020-04-23T22:39:05.826Z] 22:39:05 INFO - REFTEST INFO | Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/QNArt0tLSUekPuj9TA4Nnw/artifacts/public/build/target.crashreporter-symbols.zip
[task 2020-04-23T22:39:10.259Z] 22:39:10 INFO - REFTEST INFO | Copy/paste: /Users/cltbld/tasks/task_1587681418/fetches/minidump_stackwalk/minidump_stackwalk /var/folders/pn/nsh1xd293s16h6yswvhkj1b0000017/T/tmpjOAhIe.mozrunner/minidumps/F89E4216-9A34-4C39-9774-F34F469CD842.dmp /var/folders/pn/nsh1xd293s16h6yswvhkj1b0000017/T/tmpeNvdc4
[task 2020-04-23T22:39:18.189Z] 22:39:18 INFO - REFTEST INFO | Saved minidump as /Users/cltbld/tasks/task_1587681418/build/blobber_upload_dir/F89E4216-9A34-4C39-9774-F34F469CD842.dmp
[task 2020-04-23T22:39:18.189Z] 22:39:18 INFO - REFTEST INFO | Saved app info as /Users/cltbld/tasks/task_1587681418/build/blobber_upload_dir/F89E4216-9A34-4C39-9774-F34F469CD842.extra
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - REFTEST PROCESS-CRASH | pid: None | application crashed [@ -]
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - Crash dump filename: /var/folders/pn/nsh1xd293s16h6yswvhkj1b0000017/T/tmpjOAhIe.mozrunner/minidumps/F89E4216-9A34-4C39-9774-F34F469CD842.dmp
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - Operating system: Mac OS X
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - 10.14.5 18F132
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - CPU: amd64
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - family 6 model 69 stepping 1
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - 4 CPUs
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO -
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO - GPU: UNKNOWN
[task 2020-04-23T22:39:18.329Z] 22:39:18 INFO -
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - Crash reason: EXC_BAD_ACCESS / KERN_MEMORY_ERROR
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - Crash address: 0x5cf26b3c
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - Process uptime: 4 seconds
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO -
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - Thread 0 (crashed)
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - 0 CoreUI!-[CUIThemeFacet drawGradientInFrame:angle:alpha:operation:isFocused:keyAdjustment:context:effects:] + 0x0
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - rax = 0x0000000000000000 rdx = 0x000000000000001a
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - rcx = 0x0000000000000000 rbx = 0x000000011bdfbca0
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - rsi = 0x00007fff5cfa34ee rdi = 0x00000001268a3f10
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - rbp = 0x00007ffee98306a0 rsp = 0x00007ffee9830618
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - r8 = 0x000000011bdfbca0 r9 = 0x00000001279fe040
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - r10 = 0x000000000000002e r11 = 0x00007fff5cf26b3c
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - r12 = 0x00000001279fe040 r13 = 0x00000001268a3f10
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - r14 = 0x4036000000000000 r15 = 0x00007fff46994600
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - rip = 0x00007fff5cf26b3c
[task 2020-04-23T22:39:18.330Z] 22:39:18 INFO - Found by: given as instruction pointer in context
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - 1 CoreUI!-[CUIThemeFacet _drawSpecificRenditionKey:inFrame:context:alpha:operation:isFocused:isFlipped:effects:] + 0xcc
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rbp = 0x00007ffee9830750 rsp = 0x00007ffee98306b0
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rip = 0x00007fff5ced21e1
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - 2 CoreUI!-[CUIThemeFacet drawInFrame:alpha:operation:isFocused:context:effects:] + 0x96
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rbp = 0x00007ffee98307e0 rsp = 0x00007ffee9830760
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rip = 0x00007fff5ced20f5
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - 3 CoreUI!CUICoreThemeRenderer::DrawWindowFrameStandardNew(CUIDescriptor const*) + 0xbad
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rbp = 0x00007ffee9830ac0 rsp = 0x00007ffee98307f0
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rip = 0x00007fff5cec900d
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - 4 CoreUI!CUIRenderer::Draw(CGRect, CGContext*, __CFDictionary const*, __CFDictionary const**) + 0x6f0
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rbp = 0x00007ffee9831030 rsp = 0x00007ffee9830ad0
[task 2020-04-23T22:39:18.331Z] 22:39:18 INFO - rip = 0x00007fff5ceaea4a
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - 5 CoreUI!CUIDraw + 0x116
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rbp = 0x00007ffee98310b0 rsp = 0x00007ffee9831040
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rip = 0x00007fff5ceae32a
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - 6 AppKit!__44-[NSAppearance _drawInRect:context:options:]_block_invoke + 0x2f
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rbp = 0x00007ffee98310e0 rsp = 0x00007ffee98310c0
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rip = 0x00007fff45f35c4d
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - Found by: previous frame's frame pointer
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - 7 AppKit!-[NSCompositeAppearance _callCoreUIWithBlock:options:] + 0x16b
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rbp = 0x00007ffee98311f0 rsp = 0x00007ffee98310f0
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - rip = 0x00007fff45e22957
[task 2020-04-23T22:39:18.332Z] 22:39:18 INFO - Found by: previous frame's frame pointer

Comment 1

[Intermittent Failures Robot]

12 failures in 5582 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 10
• try: 1
• mozilla-central: 1

Platform breakdown:

• macosx1014-64-shippable: 11
• macosx1014-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1632680&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 2

[Andrew Overholt [:overholt]]

Some sort of macOS graphics shutdown issue?

Comment 3

[Markus Stange [:mstange]]

Probably the same stack as bug 1633609 - the stack in this bug is truncated, but the top pieces seem the same.
We're crashing while drawing the gradient some a window's titlebar. It does not seem to be related to shutdown.

Not sure what changed recently to make this show up now. Did the machines get updated to a different version of macOS? Are we displaying a window with a titlebar that we weren't displaying in the past?

Comment 4

[Kendall Libby [:fubar] (he/him)]

(In reply to Markus Stange [:mstange] from comment #3)

Not sure what changed recently to make this show up now. Did the machines get updated to a different version of macOS? Are we displaying a window with a titlebar that we weren't displaying in the past?

No major OS updates; maybe a hardware issue? :dhouse can help dig into the issue.

Comment 5

[:dhouse]

This was a repeat failure on one worker, t-mojave-r7-080. It does not show high temperatures or high cpu/disk/io

Comment 6

[:dhouse]

It looks like the machine crashes about once a week under normal load.
There was a crash on Apr22:

cltbld    console                   Wed Apr 22 22:40 - shutdown  (00:22)
reboot    ~                         Wed Apr 22 22:39 
cltbld    console                   Wed Apr 22 21:58 - crash  (00:41)
reboot    ~                         Wed Apr 22 21:58 
shutdown  ~                         Wed Apr 22 21:57 
cltbld    console                   Wed Apr 22 21:52 - shutdown  (00:05)
reboot    ~                         Wed Apr 22 21:52 
shutdown  ~                         Wed Apr 22 21:52 

I've collected log archives for the time of the system crash and for the core/crash for the comment #0 trace.

Comment 7

[:dhouse]

The machine was repeatedly crashing macos services. I reimaged the machine and found it reached 150C thermal sensor levels under load (compiling openssl) but did not show cores/crashes during that time.
I'll put it into a dev/staging pool and run tasks against it to see if it starts failing.

Comment 8

[Intermittent Failures Robot]

4 failures in 5423 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4

Platform breakdown:

• macosx1014-64-shippable: 3
• macosx1014-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1632680&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 9

[Firefox Bug Husbandry Bot]

Because this bug's Severity has not been changed from the default since it was filed, and it's Priority is -- (non,) indicating it has has not been previously triaged, the bug's Severity is being updated to -- (default, untriaged.)

Comment 10

[Intermittent Failures Robot]

1 failures in 3279 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1100-64-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2021-05-03&endday=2021-05-09&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 2888 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-32: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2021-06-14&endday=2021-06-20&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 3266 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-beta: 1

Platform and build breakdown:

• windows10-64-2004-shippable-qr: 1
  • opt: 1
• windows7-32-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2021-09-06&endday=2021-09-12&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 3165 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-01-31&endday=2022-02-06&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 2917 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-05-30&endday=2022-06-05&tree=all

Comment 15

[Intermittent Failures Robot]

1 failures in 3293 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-08-22&endday=2022-08-28&tree=all

Comment 16

[Intermittent Failures Robot]

2 failures in 3537 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform and build breakdown:

• windows10-32-2004-qr: 1
  • opt: 1
• windows10-64-2004-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-08-29&endday=2022-09-04&tree=all

Comment 17

[Intermittent Failures Robot]

1 failures in 3023 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows10-64-2004-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-09-12&endday=2022-09-18&tree=all

Comment 18

[Intermittent Failures Robot]

1 failures in 3626 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-qr: 1
  • debug: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2022-10-31&endday=2022-11-06&tree=all

Comment 19

[Intermittent Failures Robot]

1 failures in 3914 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform and build breakdown:

• macosx1015-64-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2023-01-23&endday=2023-01-29&tree=all

Comment 20

[Intermittent Failures Robot]

1 failures in 4077 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2023-05-08&endday=2023-05-14&tree=all

Comment 21

[Intermittent Failures Robot]

2 failures in 3578 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 1

Platform and build breakdown:

• windows11-64-2009-ccov-qr: 1
  • opt: 1
• windows11-64-2009-shippable-qr: 1
  • opt: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures/bugdetails?bug=1632680&startday=2023-05-15&endday=2023-05-21&tree=all

Comment 22

[BugBot [:suhaib / :marco/ :calixte]]

Redirect a needinfo that is pending on an inactive user to the triage owner.
:spohl, since the bug doesn't have a severity set, could you please set the severity or close the bug?

For more information, please visit BugBot documentation.