1565594 - Intermittent windows xpcshell | application crashed [@ mozilla::ipc::ProcessLink::OnChannelConnectError()]

Status: RESOLVED WORKSFORME

(Core :: IPC, defect)

Description

[Treeherder Bug Filer]

Filed by: dvarga [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=256177891&repo=mozilla-central
Full log: https://queue.taskcluster.net/v1/task/aeSxjezYTya9keuVpX66YQ/runs/0/artifacts/public/logs/live_backing.log

---

11:45:05     INFO -  TEST-START | devtools/platform/tests/unit/test_nsjsinspector.js
11:45:05     INFO -  mozcrash Saved minidump as Z:\task_1562931568\build\blobber_upload_dir\fe87c4d4-8b53-45bf-a1d6-950ecf5c89a1.dmp
11:45:05  WARNING -  PROCESS-CRASH | browser/extensions/formautofill/test/unit/heuristics/third_party/test_Macys.js | application crashed [@ mozilla::ipc::ProcessLink::OnChannelConnectError()]
11:45:05     INFO -  Crash dump filename: c:\users\task_1562931568\appdata\local\temp\xpc-other-3nbfwn\fe87c4d4-8b53-45bf-a1d6-950ecf5c89a1.dmp
11:45:05     INFO -  Operating system: Windows NT
11:45:05     INFO -                    10.0.17134
11:45:05     INFO -  CPU: amd64
11:45:05     INFO -       family 6 model 85 stepping 4
11:45:05     INFO -       8 CPUs
11:45:05     INFO -  GPU: UNKNOWN
11:45:05     INFO -  Crash reason:  EXCEPTION_ACCESS_VIOLATION_READ
11:45:05     INFO -  Crash address: 0xffffffffffffffff
11:45:05     INFO -  Assertion: Unknown assertion type 0x00000000
11:45:05     INFO -  Process uptime: 2 seconds
11:45:05     INFO -  Thread 4 (crashed)
11:45:05     INFO -   0  xul.dll!mozilla::ipc::ProcessLink::OnChannelConnectError() [MessageLink.cpp:cb2d564879e3e441c2220f389448e71e7280de6e : 336 + 0x4]
11:45:05     INFO -      rax = 0xe5e5e5e5e5e5e5e5   rdx = 0x0000000000000000
11:45:05     INFO -      rcx = 0x000001d6ff226be0   rbx = 0x000001d6ff2451c0
11:45:05     INFO -      rsi = 0x000001d6ff226be0   rdi = 0x0000009da91ffa00
11:45:05     INFO -      rbp = 0x00000000fbd10701   rsp = 0x0000009da91ff930
11:45:05     INFO -       r8 = 0x0000009da91ff898    r9 = 0x00000000fbd10701
11:45:05     INFO -      r10 = 0x0000000000000000   r11 = 0x0000000000000246
11:45:05     INFO -      r12 = 0x0000000000000000   r13 = 0x0000000000000000
11:45:05     INFO -      r14 = 0x0000009da91ffa08   r15 = 0x0000000000000000
11:45:05     INFO -      rip = 0x00007ffad25a4ebd
11:45:05     INFO -      Found by: given as instruction pointer in context
11:45:05     INFO -   1  xul.dll!nsresult mozilla::detail::RunnableMethodImpl<(anonymous namespace)::HangMonitorChild *,void ((anonymous namespace)::HangMonitorChild::*)(),0,mozilla::RunnableKind::Standard>::Run() [nsThreadUtils.h:cb2d564879e3e441c2220f389448e71e7280de6e : 1176 + 0xa]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ff970   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad2060eaa
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   2  xul.dll!MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask &&) [message_loop.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 450 + 0x1a]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ff9a0   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad2574c34
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   3  xul.dll!MessageLoop::DoWork() [message_loop.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 523 + 0x5]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ff9e0   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad25750b5
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   4  xul.dll!base::MessagePumpForIO::DoRunLoop() [message_pump_win.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 421 + 0xd]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffa50   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad25694d3
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   5  xul.dll!base::MessagePumpWin::Run(base::MessagePump::Delegate *) [message_pump_win.h:cb2d564879e3e441c2220f389448e71e7280de6e : 79 + 0x57]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffac0   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad2569747
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   6  xul.dll!MessageLoop::RunHandler() [message_loop.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 308 + 0xf]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffb20   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad25747d9
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   7  xul.dll!base::Thread::ThreadMain() [thread.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 192 + 0x4f]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffb70   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad2578dab
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   8  xul.dll!static unsigned long `anonymous namespace'::ThreadFunc(void *) [platform_thread_win.cc:cb2d564879e3e441c2220f389448e71e7280de6e : 19 + 0x6]
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffd50   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffad2569dea
11:45:05     INFO -      Found by: call frame info
11:45:05     INFO -   9  kernel32.dll!RtlpLowFragHeapAllocFromContext + 0x204
11:45:05     INFO -      rbx = 0x000001d6ff2451c0   rbp = 0x00000000fbd10701
11:45:05     INFO -      rsp = 0x0000009da91ffd80   r12 = 0x0000000000000000
11:45:05     INFO -      r13 = 0x0000000000000000   r14 = 0x0000009da91ffa08
11:45:05     INFO -      r15 = 0x0000000000000000   rip = 0x00007ffb16073034
11:45:05     INFO -      Found by: call frame info

Comment 1

[Intermittent Failures Robot]

1 failures in 4710 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-07-08&endday=2019-07-14&tree=all

Comment 2

[BugBot [:suhaib / :marco/ :calixte]]

The priority flag is not set for this bug.
:MattN, could you have a look please?

For more information, please visit auto_nag documentation.

Comment 3

[Intermittent Failures Robot]

1 failures in 4666 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-08-05&endday=2019-08-11&tree=all

Comment 4

[Intermittent Failures Robot]

1 failures in 4839 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-aarch64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-08-12&endday=2019-08-18&tree=all

Comment 6

[Intermittent Failures Robot]

1 failures in 5073 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-08-19&endday=2019-08-25&tree=all

Comment 7

[Intermittent Failures Robot]

1 failures in 4637 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1

Platform breakdown:

• windows10-64-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-09-09&endday=2019-09-15&tree=all

Comment 8

[Intermittent Failures Robot]

2 failures in 4530 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 2

Platform breakdown:

• windows10-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-09-23&endday=2019-09-29&tree=all

Comment 9

[Intermittent Failures Robot]

1 failures in 3964 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-10-14&endday=2019-10-20&tree=all

Comment 10

[Intermittent Failures Robot]

1 failures in 4850 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-esr68: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2019-11-04&endday=2019-11-10&tree=all

Comment 11

[Intermittent Failures Robot]

1 failures in 4045 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• try: 1

Platform breakdown:

• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-02-17&endday=2020-02-23&tree=all

Comment 12

[Intermittent Failures Robot]

2 failures in 5336 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• windows10-64-shippable: 1
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-03-09&endday=2020-03-15&tree=all

Comment 13

[Intermittent Failures Robot]

1 failures in 4955 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-beta: 1

Platform breakdown:

• windows10-64-devedition: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-03-16&endday=2020-03-22&tree=all

Comment 14

[Intermittent Failures Robot]

1 failures in 5015 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1

Platform breakdown:

• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-03-30&endday=2020-04-05&tree=all

Comment 15

[Intermittent Failures Robot]

2 failures in 4823 pushes (0.0 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 1

Platform breakdown:

• windows7-32-shippable: 1
• windows10-64-ccov: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-04-06&endday=2020-04-12&tree=all

Comment 16

[Intermittent Failures Robot]

3 failures in 4528 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 1
• autoland: 2

Platform breakdown:

• windows10-64-ccov: 1
• windows10-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-04-13&endday=2020-04-19&tree=all

Comment 17

[Intermittent Failures Robot]

3 failures in 5582 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 1
• mozilla-central: 2

Platform breakdown:

• windows10-64-shippable: 1
• windows10-64-ccov: 1
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-04-20&endday=2020-04-26&tree=all

Comment 18

[Intermittent Failures Robot]

3 failures in 5423 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3

Platform breakdown:

• windows10-64-ccov: 1
• windows10-64-shippable: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1565594&startday=2020-04-27&endday=2020-05-03&tree=all

Comment 21

[Jed Davis [:jld] ⟨⏰|UTC-8⟩ ⟦he/him⟧]

This is a use-after-free. Note that rax = 0xe5e5e5e5e5e5e5e5; extracting the crashing code from a recent instance and disassembling gives:

    7ffd16dcb760:       56                      push   %rsi
    7ffd16dcb761:       57                      push   %rdi
    7ffd16dcb762:       48 83 ec 28             sub    $0x28,%rsp
    7ffd16dcb766:       48 89 ce                mov    %rcx,%rsi
    7ffd16dcb769:       48 8b 41 08             mov    0x8(%rcx),%rax
    7ffd16dcb76d:       48 8b 78 30             mov    0x30(%rax),%rdi

This is Windows, so rcx would be the first argument (this): the ProcessLink was freed, the read from it returns the e5 pattern, and the read from that crashes. This is probably mChan->mMonitor here.

The crashing method is called via this ominous-looking NewNonOwningRunnableMethod. It looks like MessageLinks are owned by their MessageChannels and destroyed on the actor thread, but there's a weak reference to it from a runnable in the I/O thread's event queue. This code might have been broken ever since it was added in bug 1316473.

Comment 22

[Gian-Carlo Pascutto [:gcp]]

Jed, Nika, this signature has disappeared. Is it possible we fixed this as part of other IPC changes?

Comment 23

[Nika Layzell [:nika] (ni? for response)]

(In reply to Gian-Carlo Pascutto [:gcp] from comment #22)

Jed, Nika, this signature has disappeared. Is it possible we fixed this as part of other IPC changes?

The ProcessLink type was removed entirely in bug 1713148, so this signature can no longer exist after that point. The new replacement type of PortLink uses a more reliable system for keeping references to the MessageChannel object which hopefully shouldn't be vulnerable to this UAF bug.