1577534 - Perma tier2 LeakSanitizer | leak at NewSegment, CreateSegment, mozilla::ipc::Shmem::Alloc, mozilla::ipc::IToplevelProtocol::CreateSharedMemory

Status: RESOLVED FIXED

(Core :: Audio/Video: MediaStreamGraph, defect, P2)

Description

[Treeherder Bug Filer]

Filed by: apavel [at] mozilla.com
Parsed log: https://treeherder.mozilla.org/logviewer.html#?job_id=264060060&repo=autoland
Full log: https://queue.taskcluster.net/v1/task/NbNkvZ8pS0uSja3lhKVYzg/runs/0/artifacts/public/logs/live_backing.log

---

task 2019-08-29T15:02:50.302Z] 15:02:50 INFO - TEST-START | /mixed-content/worklet-paint-data-import/meta-csp/same-host-http/top-level/no-redirect/blockable/opt-in-blocks.https.html
[task 2019-08-29T15:02:50.306Z] 15:02:50 INFO - Closing window 119
[task 2019-08-29T15:02:50.465Z] 15:02:50 INFO - PID 18385 | -----------------------------------------------------
[task 2019-08-29T15:02:50.466Z] 15:02:50 INFO - PID 18385 | Suppressions used:
[task 2019-08-29T15:02:50.466Z] 15:02:50 INFO - PID 18385 | count bytes template
[task 2019-08-29T15:02:50.466Z] 15:02:50 INFO - PID 18385 | 27 832 nsComponentManagerImpl
[task 2019-08-29T15:02:50.466Z] 15:02:50 INFO - PID 18385 | 611 17713 libfontconfig.so
[task 2019-08-29T15:02:50.466Z] 15:02:50 INFO - PID 18385 | 1 29 libglib-2.0.so
[task 2019-08-29T15:02:50.467Z] 15:02:50 INFO - PID 18385 | -----------------------------------------------------
[task 2019-08-29T15:02:51.392Z] 15:02:51 INFO - .
[task 2019-08-29T15:02:51.392Z] 15:02:51 INFO - TEST-OK | /mixed-content/worklet-paint-data-import/meta-csp/same-host-http/top-level/no-redirect/blockable/opt-in-blocks.https.html | took 1091ms

[task 2019-08-29T15:02:55.211Z] 15:02:55 INFO - Browser exited with return code 0
[task 2019-08-29T15:02:55.211Z] 15:02:55 INFO - PROCESS LEAKS None
[task 2019-08-29T15:02:55.212Z] 15:02:55 INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 176 byte(s) leaked in 2 allocation(s).
[task 2019-08-29T15:02:55.212Z] 15:02:55 INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2019-08-29T15:02:55.212Z] 15:02:55 INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2019-08-29T15:02:55.212Z] 15:02:55 INFO - Allowed depth was 4
[task 2019-08-29T15:02:55.212Z] 15:02:55 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at NewSegment, CreateSegment, mozilla::ipc::Shmem::Alloc, mozilla::ipc::IToplevelProtocol::CreateSharedMemory
[task 2019-08-29T15:02:55.213Z] 15:02:55 INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at mozilla::layers::BufferTextureData::CreateInternal, mozilla::layers::BufferTextureData::CreateForYCbCr, mozilla::layers::TextureClient::CreateForYCbCr, mozilla::layers::YCbCrTextureClientAllocationHelper::Allocate
[task 2019-08-29T15:02:55.214Z] 15:02:55 INFO - Closing logging queue

Comment 1

[Andreea Pavel [:apavel]]

Backfill: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=pending%2Crunning%2Csuccess%2Ctestfailed%2Cbusted%2Cexception&fromchange=3d205e901e3af48a6ef6b1f6e183516b83efda9f&searchStr=linux%2Cx64%2Casan%2Cweb%2Cplatform%2Ctests%2Ctest-linux64-asan%2Fopt-web-platform-tests-e10s-14%2Cw%28wpt14%29&selectedJob=264081137

Comment 2

[Cosmin Sabou [:CosminS]]

Andreas, could you please look into these leaks? Thank you.
https://treeherder.mozilla.org/#/jobs?repo=autoland&group_state=expanded&resultStatus=success%2Ctestfailed%2Cbusted%2Cexception&classifiedState=unclassified&searchStr=linux%2Cx64%2Casan%2Cweb%2Cplatform%2Ctests%2Ctest-linux64-asan%2Fopt-web-platform-tests-e10s-4%2Cw%28wpt4%29&tochange=212bfffe0a67c43e2c622a74b0009f182bbb6ffc&fromchange=8edbf8fe48bfd2daeff36e5a6716e5d8c6cc3f69&selectedJob=264078335

Comment 3

[Andreas Pehrson [:pehrsons]]

I can see where these shmem buffers are allocated -- but the question is where they're held when we leak. I'll have to dig deeper tomorrow.

Comment 4

[Intermittent Failures Robot]

83 failures in 1013 pushes (0.082 failures/push) were associated with this bug yesterday.

Repository breakdown:

• autoland: 74
• try: 3
• mozilla-central: 4
• mozilla-inbound: 2

Platform breakdown:

• linux64: 83

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-08-29&endday=2019-08-29&tree=all

Comment 5

[Andreas Pehrson [:pehrsons]]

This is the relevant part of the stack when releasing that gfx resource, from a try run where I elevated the gfx error:

[task 2019-08-29T19:17:01.714Z] 19:17:01     INFO - PID 18332 |     #23 0x7f8b7e1c33ed in mozilla::layers::SharedPlanarYCbCrImage::~SharedPlanarYCbCrImage() /builds/worker/workspace/build/src/gfx/layers/ipc/SharedPlanarYCbCrImage.cpp:36:51
[task 2019-08-29T19:17:01.724Z] 19:17:01     INFO - PID 18332 |     #24 0x7f8b81efa3be in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/ImageContainer.h:194:3
[task 2019-08-29T19:17:01.726Z] 19:17:01     INFO - PID 18332 |     #25 0x7f8b81efa3be in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:48
[task 2019-08-29T19:17:01.727Z] 19:17:01     INFO - PID 18332 |     #26 0x7f8b81efa3be in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:373
[task 2019-08-29T19:17:01.729Z] 19:17:01     INFO - PID 18332 |     #27 0x7f8b81efa3be in ~RefPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:79
[task 2019-08-29T19:17:01.731Z] 19:17:01     INFO - PID 18332 |     #28 0x7f8b81efa3be in ~VideoFrame /builds/worker/workspace/build/src/dom/media/VideoSegment.cpp:30
[task 2019-08-29T19:17:01.732Z] 19:17:01     INFO - PID 18332 |     #29 0x7f8b81efa3be in ~VideoChunk /builds/worker/workspace/build/src/dom/media/VideoSegment.h:65
[task 2019-08-29T19:17:01.734Z] 19:17:01     INFO - PID 18332 |     #30 0x7f8b81efa3be in Destruct /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:554
[task 2019-08-29T19:17:01.735Z] 19:17:01     INFO - PID 18332 |     #31 0x7f8b81efa3be in DestructRange /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:2224
[task 2019-08-29T19:17:01.737Z] 19:17:01     INFO - PID 18332 |     #32 0x7f8b81efa3be in ClearAndRetainStorage /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1329
[task 2019-08-29T19:17:01.738Z] 19:17:01     INFO - PID 18332 |     #33 0x7f8b81efa3be in ~nsTArray_Impl /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:910
[task 2019-08-29T19:17:01.740Z] 19:17:01     INFO - PID 18332 |     #34 0x7f8b81efa3be in mozilla::MediaSegmentBase<mozilla::VideoSegment, mozilla::VideoChunk>::~MediaSegmentBase() /builds/worker/workspace/build/src/dom/media/MediaSegment.h:235
[task 2019-08-29T19:17:01.741Z] 19:17:01     INFO - PID 18332 |     #35 0x7f8b81eea3ed in ~VideoSegment /builds/worker/workspace/build/src/dom/media/VideoSegment.cpp:112:32
[task 2019-08-29T19:17:01.743Z] 19:17:01     INFO - PID 18332 |     #36 0x7f8b81eea3ed in mozilla::VideoSegment::~VideoSegment() /builds/worker/workspace/build/src/dom/media/VideoSegment.cpp:112
[task 2019-08-29T19:17:01.744Z] 19:17:01     INFO - PID 18332 |     #37 0x7f8b82257dbc in ~nsAutoPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/nsAutoPtr.h:65:18
[task 2019-08-29T19:17:01.746Z] 19:17:01     INFO - PID 18332 |     #38 0x7f8b82257dbc in ~Track /builds/worker/workspace/build/src/dom/media/StreamTracks.h:70
[task 2019-08-29T19:17:01.747Z] 19:17:01     INFO - PID 18332 |     #39 0x7f8b82257dbc in ~nsAutoPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/nsAutoPtr.h:65
[task 2019-08-29T19:17:01.748Z] 19:17:01     INFO - PID 18332 |     #40 0x7f8b82257dbc in Destruct /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:554
[task 2019-08-29T19:17:01.749Z] 19:17:01     INFO - PID 18332 |     #41 0x7f8b82257dbc in DestructRange /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:2224
[task 2019-08-29T19:17:01.750Z] 19:17:01     INFO - PID 18332 |     #42 0x7f8b82257dbc in ClearAndRetainStorage /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1329
[task 2019-08-29T19:17:01.750Z] 19:17:01     INFO - PID 18332 |     #43 0x7f8b82257dbc in Clear /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1805
[task 2019-08-29T19:17:01.750Z] 19:17:01     INFO - PID 18332 |     #44 0x7f8b82257dbc in mozilla::StreamTracks::Clear() /builds/worker/workspace/build/src/dom/media/StreamTracks.cpp:98
[task 2019-08-29T19:17:01.765Z] 19:17:01     INFO - PID 18332 |     #45 0x7f8b822104d3 in mozilla::MediaStream::DestroyImpl() /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:1985:11
[task 2019-08-29T19:17:01.765Z] 19:17:01     INFO - PID 18332 |     #46 0x7f8b82214c3f in mozilla::SourceMediaStream::DestroyImpl() /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:2412:16
[task 2019-08-29T19:17:01.765Z] 19:17:01     INFO - PID 18332 |     #47 0x7f8b82239f1c in mozilla::MediaStream::Destroy()::Message::Run() /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:1999:16
[task 2019-08-29T19:17:01.767Z] 19:17:01     INFO - PID 18332 |     #48 0x7f8b82206cd9 in mozilla::MediaStreamGraphImpl::RunMessagesInQueue() /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:1167:20
[task 2019-08-29T19:17:01.770Z] 19:17:01     INFO - PID 18332 |     #49 0x7f8b8220aed6 in mozilla::MediaStreamGraphImpl::OneIterationImpl(long) /builds/worker/workspace/build/src/dom/media/MediaStreamGraph.cpp:1400:3
[task 2019-08-29T19:17:01.770Z] 19:17:01     INFO - PID 18332 |     #50 0x7f8b81f60be7 in mozilla::AudioCallbackDriver::DataCallback(float const*, float*, long) /builds/worker/workspace/build/src/dom/media/GraphDriver.cpp:882:36

Comment 6

[Andreas Pehrson [:pehrsons]]

It looks like what leads to this is a window being closed and a CubebOperation (driver switch/start ?) taking some time to finish, leading us to iterate the graph later than expected:

[task 2019-08-29T19:16:59.974Z] 19:16:59     INFO - TEST-START | /mediacapture-streams/MediaStream-finished-add.https.html
[task 2019-08-29T19:16:59.974Z] 19:16:59     INFO - Closing window 61
[task 2019-08-29T19:17:00.100Z] 19:17:00     INFO - PID 18332 | SharedThreadPool in xpcom-shutdown-threads. Waiting for pools "CubebOperation"
...
[task 2019-08-29T19:17:00.753Z] 19:17:00     INFO - PID 18332 | Crash Annotation GraphicsCriticalError: |[C0][GFX1]: Texture deallocated too late during shutdown (t=4.40205) [GFX1]: Texture deallocated too late during shutdown
[task 2019-08-29T19:17:00.753Z] 19:17:00     INFO - PID 18332 | Assertion failure: false (Texture deallocated too late during shutdown), at /builds/worker/workspace/build/src/gfx/layers/client/TextureClient.cpp:481

Having a MediaStream around in xpcom-shutdown-threads is unexpected.

MediaStreamGraph has an AsyncShutdownBlocker -- are we removing that before all streams are removed?

Comment 7

[Andreas Pehrson [:pehrsons]]

Right, if the blocker is notified of shutdown after the last stream has been removed on main thread (graph removed from gGraphs), that graph won't be notified, and shutdown progresses too soon.

Comment 8

[Intermittent Failures Robot]

96 failures in 1088 pushes (0.088 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 68
• mozilla-central: 14
• mozilla-inbound: 11
• try: 3

Platform breakdown:

• linux64: 96

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-08-30&endday=2019-08-30&tree=all

Comment 9

[Andreea Pavel [:apavel]]

Hi Andreas. seeing that this has a very high occurrence rate, can you give a time estimate for a fix?

Comment 10

[Intermittent Failures Robot]

41 failures in 191 pushes (0.215 failures/push) were associated with this bug yesterday.

Repository breakdown:

• mozilla-inbound: 9
• autoland: 23
• mozilla-central: 6
• try: 3

Platform breakdown:

• linux64: 41

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-08-31&endday=2019-08-31&tree=all

Comment 11

[Karl Tomlinson (:karlt)]

I wonder whether it is possible to create a ShutdownTicket when we remove the graph from gGraphs, which I assume is when we know the graph will be shutdown, but that is assuming that remaining ports will be removed soon. Maybe it would be more consistent to only remove from gGraphs when there are neither streams nor ports.

Comment 12

[Intermittent Failures Robot]

251 failures in 5047 pushes (0.05 failures/push) were associated with this bug in the last 7 days.

This is the #3 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 184
• try: 14
• mozilla-inbound: 26
• mozilla-central: 27

Platform breakdown:

• linux64: 250
• linux32-shippable: 1

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-08-26&endday=2019-09-01&tree=all

Comment 13

[Andreas Pehrson [:pehrsons]]

I expect to get a fix in today. I wrote up a patch on friday that hit some issues but now largely works. It does a bit much perhaps and is prone to create leaks. I'll write up a simpler thing today that focuses on MediaStreamGraph rather than the general ShutdownBlocker/ShutdownTicket mechanisms.

Comment 14

[Andreas Pehrson [:pehrsons]]

Attachment: Bug 1577534 - Give MediaStreamGraphs individual ShutdownBlockers. r?karlt

Depends on D44369

Comment 15

[Andreas Pehrson [:pehrsons]]

https://treeherder.mozilla.org/#/jobs?repo=try&revision=d78007dd031139f3422c3d4095bfbf19e6b9eb3f

Comment 16

[Andreas Pehrson [:pehrsons]]

This will have to wait until tomorrow, because of the timezone difference to the reviewer. padenot was not able to review today.

Comment 17

[Pulsebot]

Pushed by [e-mail]:
https://hg.mozilla.org/integration/autoland/rev/4feb20ebb67d
Give MediaStreamGraphs individual ShutdownBlockers. r=karlt

Comment 18

[Intermittent Failures Robot]

71 failures in 500 pushes (0.142 failures/push) were associated with this bug yesterday.

** This test has failed more than 150 times in the last 21 days. It should be disabled until it can be fixed. **

Repository breakdown:

• autoland: 36
• mozilla-beta: 9
• mozilla-central: 8
• mozilla-inbound: 10
• try: 8

Platform breakdown:

• linux64: 71

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-09-02&endday=2019-09-02&tree=all

Comment 19

[Razvan Maries]

https://hg.mozilla.org/mozilla-central/rev/4feb20ebb67d

Comment 20

[Andreas Pehrson [:pehrsons]]

Comment on attachment 9089870 [details]
Bug 1577534 - Give MediaStreamGraphs individual ShutdownBlockers. r?karlt

Beta/Release Uplift Approval Request

• User impact if declined: Possible small leak if a content process is shut down at the wrong time. High frequency in CI.
• Is this code covered by automated tests?: Yes
• Has the fix been verified in Nightly?: No
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Slight functionality refactor that looks fairly large. No timing or threading changes which is why I consider it low risk.
• String changes made/needed:

Comment 22

[Liz Henry (:lizzard) (relman/hg->git project)]

Comment on attachment 9089870 [details]
Bug 1577534 - Give MediaStreamGraphs individual ShutdownBlockers. r?karlt

Fix for shutdown memory leak, OK for uplift for beta 4.

Comment 23

[Bogdan Tara[:bogdan_tara | bogdant]]

https://hg.mozilla.org/releases/mozilla-beta/rev/02bc71fde278

Comment 24

[Intermittent Failures Robot]

94 failures in 4408 pushes (0.021 failures/push) were associated with this bug in the last 7 days.

This is the #12 most frequent failure this week.

** This failure happened more than 75 times this week! Resolving this bug is a very high priority. **

** Try to resolve this bug as soon as possible. If unresolved for 1 week, the affected test(s) may be disabled. **

Repository breakdown:

• autoland: 36
• mozilla-beta: 9
• mozilla-central: 14
• mozilla-inbound: 14
• try: 21

Platform breakdown:

• linux64: 94

For more details, see:
https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?bug=1577534&startday=2019-09-02&endday=2019-09-08&tree=all