1566294 - Perma [Tier 2] PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::ipc::FatalError(char const*, bool)]

Status: RESOLVED FIXED

(Core Graveyard :: Web Replay, defect)

Description

[Treeherder Bug Filer]

Filed by: malexandru [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=256632590&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/dmjifhqqTSusQsVTtQjoHA/runs/0/artifacts/public/logs/live_backing.log

---

23:38:55 INFO - GECKO(1637) | Spew[1809]: WARNING: Repeated accesses to the same dirty address 0x111a0dba0
23:38:57 INFO - GECKO(1637) | IPDL protocol error: Error unknown callback
23:38:57 INFO - GECKO(1637) | MEMORY STAT | vsize 7890MB | residentFast 564MB | heapAllocated 219MB
23:38:57 INFO - TEST-OK | devtools/client/webreplay/mochitest/browser_rr_inspector-03.js | took 10499ms
23:38:57 INFO - checking window state
23:38:57 INFO - GECKO(1637) | JavaScript error: resource://testing-common/PromiseTestUtils.jsm, line 112: uncaught exception: Object
23:38:57 INFO - Console message: [JavaScript Error: "uncaught exception: Object" {file: "resource://testing-common/PromiseTestUtils.jsm" line: 112}]
<...>
23:39:01 INFO - mozcrash Downloading symbols from: https://queue.taskcluster.net/v1/task/cK2felo9QgilR3TWTVI_vA/artifacts/public/build/target.crashreporter-symbols.zip
23:39:05 INFO - mozcrash Copy/paste: /Users/cltbld/tasks/task_1563233597/build/macosx64-minidump_stackwalk /var/folders/0d/t2ch6yv958q7bvm9fbndb8qc000017/T/tmpOZfMhe.mozrunner/minidumps/D26D741B-E14F-478B-A150-D872887B9E60.dmp /var/folders/0d/t2ch6yv958q7bvm9fbndb8qc000017/T/tmpQ25q7v
23:39:09 INFO - mozcrash Saved minidump as /Users/cltbld/tasks/task_1563233597/build/blobber_upload_dir/D26D741B-E14F-478B-A150-D872887B9E60.dmp
23:39:10 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ mozilla::ipc::FatalError(char const*, bool)]
23:39:10 INFO - Crash dump filename: /var/folders/0d/t2ch6yv958q7bvm9fbndb8qc000017/T/tmpOZfMhe.mozrunner/minidumps/D26D741B-E14F-478B-A150-D872887B9E60.dmp
23:39:10 INFO - Operating system: Mac OS X
23:39:10 INFO - 10.14.5 18F132
23:39:10 INFO - CPU: amd64
23:39:10 INFO - family 6 model 69 stepping 1
23:39:10 INFO - 4 CPUs
23:39:10 INFO -
23:39:10 INFO - GPU: UNKNOWN
23:39:10 INFO -
23:39:10 INFO - Crash reason: EXC_BAD_ACCESS / KERN_INVALID_ADDRESS
23:39:10 INFO - Crash address: 0x0
23:39:10 INFO - Process uptime: 118 seconds
23:39:10 INFO -
23:39:10 INFO - Thread 0 (crashed)
23:39:10 INFO - 0 XUL!mozilla::ipc::FatalError(char const*, bool) [ProtocolUtils.cpp:54d9810b02ae4da2b278c06576f4d9e4438be7e5 : 264 + 0xe]
23:39:10 INFO - rax = 0x00007ffeedc4412c rdx = 0x000000000000000a
23:39:10 INFO - rcx = 0x00000001023a7de0 rbx = 0x000000010eff0619
23:39:10 INFO - rsi = 0x000000010efdf17d rdi = 0x00007ffeedc4412c
23:39:10 INFO - rbp = 0x00007ffeedc44180 rsp = 0x00007ffeedc44100
23:39:10 INFO - r8 = 0x0000000000000000 r9 = 0x0000000000000000
23:39:10 INFO - r10 = 0x0000000000000000 r11 = 0x00007ffddec64fd2
23:39:10 INFO - r12 = 0x0000000102536ac0 r13 = 0x0000000102f61830
23:39:10 INFO - r14 = 0x0000000000000000 r15 = 0x0000000102f52020
23:39:10 INFO - rip = 0x0000000109d09721
23:39:10 INFO - Found by: given as instruction pointer in context
23:39:10 INFO - 1 XUL!mozilla::ipc::IProtocol::HandleFatalError(char const*) const [ProtocolUtils.cpp:54d9810b02ae4da2b278c06576f4d9e4438be7e5 : 495 + 0x13]
23:39:10 INFO - rbp = 0x00007ffeedc44190 rsp = 0x00007ffeedc44190
23:39:10 INFO - rip = 0x0000000109d0aeac
23:39:10 INFO - Found by: previous frame's frame pointer
23:39:10 INFO - 2 XUL!mozilla::dom::PContentChild::OnMessageReceived(IPC::Message const&) [PContentChild.cpp: : 12076 + 0x10]
23:39:10 INFO - rbp = 0x00007ffeedc44f80 rsp = 0x00007ffeedc441a0
23:39:10 INFO - rip = 0x0000000109e09b00
23:39:10 INFO - Found by: previous frame's frame pointer
23:39:10 INFO - 3 XUL!mozilla::ipc::MessageChannel::DispatchMessage(IPC::Message&&) [MessageChannel.cpp:54d9810b02ae4da2b278c06576f4d9e4438be7e5 : 2158 + 0xd]
23:39:10 INFO - rbp = 0x00007ffeedc45050 rsp = 0x00007ffeedc44f90
23:39:10 INFO - rip = 0x0000000109d03963
23:39:10 INFO - Found by: previous frame's frame pointer
23:39:10 INFO - 4 XUL!mozilla::ipc::MessageChannel::MessageTask::Run() [MessageChannel.cpp:54d9810b02ae4da2b278c06576f4d9e4438be7e5 : 1970 + 0x15f]
23:39:10 INFO - rbp = 0x00007ffeedc450a0 rsp = 0x00007ffeedc45060
23:39:10 INFO - rip = 0x0000000109d04d18
23:39:10 INFO - Found by: previous frame's frame pointer

Comment 1

[Alexandru Michis [:malexandru]]

Nika, this seems to have started with your changes in Bug 1563604, could you please look into it?

Comment 2

[Intermittent Failures Robot]

19 failures in 792 pushes (0.024 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 19

Platform breakdown:

• macosx1014-64-shippable: 19

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566294&startday=2019-07-15&endday=2019-07-15&tree=all

Comment 3

[Nika Layzell [:nika] (ni? for response)]

These test failures are all in webreplay which does some pretty weird stuff with IPC. 302 :bhackett to look into it.

Comment 4

[Intermittent Failures Robot]

87 failures in 816 pushes (0.107 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 80
• mozilla-central: 2
• mozilla-inbound: 5

Platform breakdown:

• macosx1014-64-shippable: 87

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566294&startday=2019-07-16&endday=2019-07-16&tree=all

Comment 5

[Brian Hackett [Laid off!]]

Attachment: Bug 1566294 - Use distinct sequence numbers for messages sent from middleman processes, r=nika.

Comment 6

[Andreea Pavel [:apavel]]

Hi Brian. The dt-wr job seems to fail each time it runs, either with this bug or with other ones:

https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=testfailed%2Cbusted%2Cexception&searchStr=os%2Cx%2C10.14%2Cshippable%2Copt%2Cmochitests%2Ctest-macosx1014-64-shippable%2Fopt-mochitest-devtools-webreplay-e10s%2Cm%28dt-wr%29&fromchange=07572fb43cf7b822c09d90a8b39f1280b6d531ce&tochange=64d95a85657b8200007ed62d9672bf6009f01aed&selectedJob=257001690

Any suggestion on how to get the job green?

Comment 7

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/mozilla-inbound/rev/905cb53bd118
Use distinct sequence numbers for messages sent from middleman processes, r=nika.

Comment 8

[Intermittent Failures Robot]

95 failures in 945 pushes (0.101 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 88
• mozilla-central: 5
• mozilla-inbound: 2

Platform breakdown:

• macosx1014-64-shippable: 95

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566294&startday=2019-07-17&endday=2019-07-17&tree=all

Comment 9

[Alexandru Michis [:malexandru]]

https://hg.mozilla.org/mozilla-central/rev/905cb53bd118

Comment 10

[Intermittent Failures Robot]

37 failures in 811 pushes (0.046 failures/push) were associated with this bug yesterday.

Repository breakdown:

• try: 1
• autoland: 34
• mozilla-central: 2

Platform breakdown:

• macosx1014-64-shippable: 37

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566294&startday=2019-07-18&endday=2019-07-18&tree=all

Comment 11

[Intermittent Failures Robot]

240 failures in 4298 pushes (0.056 failures/push) were associated with this bug in the last 7 days.

This is the #1 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 221
• mozilla-central: 9
• mozilla-inbound: 7
• try: 3

Platform breakdown:

• macosx1014-64-shippable: 239
• macosx1014-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1566294&startday=2019-07-15&endday=2019-07-21&tree=all