1653060 - Intermittent leakcheck | rdd 3544 bytes leaked (AbstractThread, ActorLifecycleProxy, CondVar, IPC::Channel, Mutex, ...)

Status: RESOLVED FIXED

(Core :: Audio/Video: Playback, defect, P1)

Description

[Treeherder Bug Filer]

Filed by: btara [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=309830807&repo=autoland
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/Cnz-C20qSfqV3BlBQP8I0Q/runs/0/artifacts/public/logs/live_backing.log

---

...
[task 2020-07-15T10:11:42.336Z] 10:11:42     INFO - == BloatView: ALL (cumulative) LEAK AND BLOAT STATISTICS, rdd process 148
[task 2020-07-15T10:11:42.337Z] 10:11:42     INFO - 
[task 2020-07-15T10:11:42.338Z] 10:11:42     INFO -      |<----------------Class--------------->|<-----Bytes------>|<----Objects---->|
[task 2020-07-15T10:11:42.338Z] 10:11:42     INFO -      |                                      | Per-Inst   Leaked|   Total      Rem|
[task 2020-07-15T10:11:42.339Z] 10:11:42     INFO -    0 |TOTAL                                 |       23     3544|    5739       16|
[task 2020-07-15T10:11:42.340Z] 10:11:42     INFO -    1 |AbstractThread                        |       40       40|       3        1|
[task 2020-07-15T10:11:42.340Z] 10:11:42     INFO -    2 |ActorLifecycleProxy                   |       32       32|       5        1|
[task 2020-07-15T10:11:42.341Z] 10:11:42     INFO -   14 |CondVar                               |       64      128|      23        2|
[task 2020-07-15T10:11:42.342Z] 10:11:42     INFO -   25 |IPC::Channel                          |        8        8|       4        1|
[task 2020-07-15T10:11:42.342Z] 10:11:42     INFO -   45 |Mutex                                 |       80      240|     277        3|
[task 2020-07-15T10:11:42.343Z] 10:11:42     INFO -   61 |PRemoteDecoderManagerParent           |      688      688|       1        1|
[task 2020-07-15T10:11:42.344Z] 10:11:42     INFO -   74 |RefCountedMonitor                     |      152      152|       4        1|
[task 2020-07-15T10:11:42.344Z] 10:11:42     INFO -   76 |RemoteDecoderManagerParent            |      736     1472|       2        2|
[task 2020-07-15T10:11:42.345Z] 10:11:42     INFO -   91 |StoreRef                              |       16       16|       4        1|
[task 2020-07-15T10:11:42.345Z] 10:11:42     INFO -   97 |TaskQueue                             |      336      336|       2        1|
[task 2020-07-15T10:11:42.346Z] 10:11:42     INFO -  128 |ipc::MessageChannel                   |      392      392|       4        1|
[task 2020-07-15T10:11:42.346Z] 10:11:42     INFO -  129 |ipc::MessageChannel::DispatchOnChannel|       40       40|       4        1|
[task 2020-07-15T10:11:42.346Z] 10:11:42     INFO - 
[task 2020-07-15T10:11:42.346Z] 10:11:42     INFO - nsTraceRefcnt::DumpStatistics: 151 entries
[task 2020-07-15T10:11:42.347Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 AbstractThread
[task 2020-07-15T10:11:42.347Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 ActorLifecycleProxy
[task 2020-07-15T10:11:42.347Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 2 CondVar
[task 2020-07-15T10:11:42.347Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 IPC::Channel
[task 2020-07-15T10:11:42.348Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 3 Mutex
[task 2020-07-15T10:11:42.348Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 PRemoteDecoderManagerParent
[task 2020-07-15T10:11:42.348Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 RefCountedMonitor
[task 2020-07-15T10:11:42.348Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 2 RemoteDecoderManagerParent
[task 2020-07-15T10:11:42.349Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 StoreRef
[task 2020-07-15T10:11:42.349Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 TaskQueue
[task 2020-07-15T10:11:42.349Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 ipc::MessageChannel
[task 2020-07-15T10:11:42.349Z] 10:11:42     INFO - TEST-INFO | leakcheck | rdd leaked 1 ipc::MessageChannel::DispatchOnChannel
[task 2020-07-15T10:11:42.350Z] 10:11:42     INFO - TEST-UNEXPECTED-FAIL | leakcheck | rdd 3544 bytes leaked (AbstractThread, ActorLifecycleProxy, CondVar, IPC::Channel, Mutex, ...)
[task 2020-07-15T10:11:42.350Z] 10:11:42     INFO - 
...

Comment 1

[Intermittent Failures Robot]

6 failures in 4014 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 4
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 4
• windows10-64: 1
• linux1804-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-07-13&endday=2020-07-19&tree=all

Comment 2

[Intermittent Failures Robot]

8 failures in 4310 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8

Platform breakdown:

• linux1804-64: 1
• windows10-64: 2
• windows10-64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-07-20&endday=2020-07-26&tree=all

Comment 3

[Intermittent Failures Robot]

11 failures in 4289 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 8
• mozilla-central: 2
• try: 1

Platform breakdown:

• windows10-64-qr: 9
• windows10-64: 2

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-07-27&endday=2020-08-02&tree=all

Comment 4

[Andrew McCreight (out of office until 8/21) [:mccr8]]

RemoteDecoderManagerParent

Comment 5

[Intermittent Failures Robot]

9 failures in 4618 pushes (0.002 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• mozilla-central: 3
• autoland: 6

Platform breakdown:

• windows10-64-qr: 8
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-08-03&endday=2020-08-09&tree=all

Comment 6

[Intermittent Failures Robot]

5 failures in 3601 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 3
• mozilla-central: 2

Platform breakdown:

• windows10-64-qr: 4
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-08-10&endday=2020-08-16&tree=all

Comment 7

[Intermittent Failures Robot]

11 failures in 4100 pushes (0.003 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 9
• mozilla-central: 1
• try: 1

Platform breakdown:

• windows10-64: 4
• windows10-64-qr: 7

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-08-17&endday=2020-08-23&tree=all

Comment 8

[Kelsey Gilbert [:jgilbert]]

This seems to perma-fail here:
https://treeherder.mozilla.org/#/jobs?repo=try&author=jgilbert%40mozilla.com&selectedTaskRun=KesinwHCQhypjZwhUafHYA.0

Comment 9

[Intermittent Failures Robot]

5 failures in 4046 pushes (0.001 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 5

Platform breakdown:

• windows10-64-qr: 3
• windows10-64: 1
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-08-24&endday=2020-08-30&tree=all

Comment 10

[Intermittent Failures Robot]

16 failures in 836 pushes (0.019 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 13
• mozilla-central: 3

Platform breakdown:

• macosx1014-64: 10
• windows10-64-qr: 5
• windows7-32: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-09-02&endday=2020-09-02&tree=all

Comment 11

[Kelsey Gilbert [:jgilbert]]

Attachment: Bug 1653060 - Increase RDD leak threshold to 4000 for now.

Comment 12

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/056bbc57ca7c
Increase RDD leak threshold to 4000 for now. r=mccr8

Comment 13

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/mozilla-central/rev/056bbc57ca7c

Comment 14

[Intermittent Failures Robot]

38 failures in 2984 pushes (0.013 failures/push) were associated with this bug in the last 7 days.

This is the #23 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• mozilla-central: 9
• autoland: 29

Platform breakdown:

• windows7-32: 11
• windows10-64-qr: 9
• macosx1014-64: 17
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-08-31&endday=2020-09-06&tree=all

Comment 15

[Jean-Yves Avenard [:jya]]

Background information on what is happening here.

The parent process control the RDDParent via the PRDD protocol.
Then we have content processes potentially using a RemoteDecoderManagerChild/Parent via the PRemoteDecoderManager protocol.

When shutting down, the RDDChild will be shutdown when XPCOM shuts down, which will instruct the RDDParent to shutdown the RDD process.

This can happen before each content process have completed shutdown and closed their respective PRemoteDecoderManager channel.
All those RemoteDecoderManagerParent waiting to be closed will be reported as leaking. A RemoteDecoderManagerParent manage a taskqueue, and an array of images.

The GPU process is leaking RemoteDecoderManagerParent in exactly the same fashion, however it has a threshold set to 10kB and so the leaks aren't apparent. But they are definitely there.

We would need to have a mechanism similar to the MediaShutdownMonitor in the RDD and GPU process, so that even when the GPUParent/RDDParent receive an intstruction to shutdown their process, they won't act on it until all registered RemoteDecoderManagerParent actors have been closed.

Comment 16

[Intermittent Failures Robot]

32 failures in 3410 pushes (0.009 failures/push) were associated with this bug in the last 7 days.

This is the #26 most frequent failure this week.

** This failure happened more than 30 times this week! Resolving this bug is a high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 2 weeks, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 25
• mozilla-central: 7

Platform breakdown:

• macosx1014-64: 27
• windows10-64-qr: 5

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-09-07&endday=2020-09-13&tree=all

Comment 18

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1653060 - P1. Revert "Increase RDD leak threshold to 4000 for now. r=mccr8"

This reverts commit 056bbc57ca7c

Comment 19

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1653060 - P2. Ensure the RDD process gets shutdown after content processes. r?mjf

The RDD process gets shutdown following a NS_XPCOM_SHUTDOWN_OBSERVER_ID notification.
Notifications are processed in LIFO order, since the RDD process is started on demand it would have typically be registered after a content process.
We must ensure that the RDD get shutdown after all content processes so that it can receive notifications that the RemoteDecoderManagerChilds are shutting down.

Depends on D90484

Comment 20

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1653060 - P3. Use nsCOMPtr. r?mattwoodrow

Depends on D90485

Comment 21

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1653060 - P4. Wait until all MediaRemoteDecoderManagerParent have closed before killing process. r?mattwoodrow.

We unfortunately can't use the AsyncShutdownService in either the GPU or RDD process.

So we add a little utility class AsyncBlockers that will resolve its promise once all services have deregistered from it.

We use it to temporily suspend the RDDParent or GPUParent from killing the process, up to 10s.

This allows for cleaner shutdown as the parent process doesn't guarantee the order in which processes are killed (even though it should).

Depends on D90486

Comment 22

[Jean-Yves Avenard [:jya]]

Attachment: Bug 1653060 - P5. Ensure no task gets dispatched after shutdown. r?mattwoodrow

And fix thread-safety access to sRemoteDecoderManagerChildThread static while at it.

Depends on D90487

Comment 23

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/68c5b2c56f3f
P1. Revert "Increase RDD leak threshold to 4000 for now. r=mccr8"
https://hg.mozilla.org/integration/autoland/rev/a38425c96709
P2. Ensure the RDD process gets shutdown after content processes. r=mjf
https://hg.mozilla.org/integration/autoland/rev/6943102ffe2a
P3. Use nsCOMPtr. r=mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/d143ac59991f
P4. Wait until all MediaRemoteDecoderManagerParent have closed before killing process. r=mattwoodrow.

Comment 24

[Cristian Brindusan [:cbrindusan]]

Backed out 4 changesets (bug 1653060) for causing mochitest assertion failures.
https://hg.mozilla.org/integration/autoland/rev/bc025e45723c7374021ef823168683103f5f1eb9

Push with failures:
https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&revision=d143ac59991ff804d792008b64383c6997078c94&selectedTaskRun=EcTYnAAnTYKH7DUh5mIctg.0

Failure log:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=316062508&repo=autoland&lineNumber=1749

Comment 25

[Jean-Yves Avenard [:jya]]

For some reasons, P5 didn't get push which fixed those assertions.

Comment 26

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/ef7eca1879d9
P1. Revert "Increase RDD leak threshold to 4000 for now. r=mccr8"
https://hg.mozilla.org/integration/autoland/rev/a8124dd6945d
P2. Ensure the RDD process gets shutdown after content processes. r=mjf
https://hg.mozilla.org/integration/autoland/rev/22b8a193cae1
P3. Use nsCOMPtr. r=mattwoodrow
https://hg.mozilla.org/integration/autoland/rev/fe83737b9acf
P4. Wait until all MediaRemoteDecoderManagerParent have closed before killing process. r=mattwoodrow.
https://hg.mozilla.org/integration/autoland/rev/3695534b2938
P5. Ensure no task gets dispatched after shutdown. r=mattwoodrow

Comment 27

[Cristian Brindusan [:cbrindusan]]

https://hg.mozilla.org/mozilla-central/rev/ef7eca1879d9
https://hg.mozilla.org/mozilla-central/rev/a8124dd6945d
https://hg.mozilla.org/mozilla-central/rev/22b8a193cae1
https://hg.mozilla.org/mozilla-central/rev/fe83737b9acf
https://hg.mozilla.org/mozilla-central/rev/3695534b2938

Comment 28

[Intermittent Failures Robot]

25 failures in 3760 pushes (0.007 failures/push) were associated with this bug in the last 7 days.

Repository breakdown:

• autoland: 22
• mozilla-central: 3

Platform breakdown:

• macosx1014-64: 22
• windows10-64-qr: 2
• windows10-64: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1653060&startday=2020-09-14&endday=2020-09-20&tree=all