Closed Bug 1477492 Opened Last year Closed Last year

Perma-failing LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys when Gecko 63 merges to Beta on 2018-08-27

Categories

(Core :: DOM: Core & HTML, defect, P2)

defect

Tracking

()

VERIFIED FIXED
mozilla63
Tracking Status
thunderbird_esr52 --- unaffected
thunderbird_esr60 --- unaffected
firefox-esr52 --- unaffected
firefox-esr60 --- unaffected
firefox61 --- unaffected
firefox62 --- unaffected
firefox63 + verified

People

(Reporter: ccoroiu, Assigned: bryce)

References

Details

Attachments

(1 file)

Central as beta simulation: 
https://treeherder.mozilla.org/?failureClassificationId=3#/jobs?repo=try&revision=568e142efb2341632a696a3431b45e9dfd30cb12&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=189324012

Failure log: 
https://treeherder.mozilla.org/logviewer.html#?job_id=189324012&repo=try&lineNumber=7299

[task 2018-07-21T12:51:18.142Z] 12:51:18     INFO - PID 11652 | Indirect leak of 24 byte(s) in 1 object(s) allocated from:
[task 2018-07-21T12:51:18.142Z] 12:51:18     INFO - PID 11652 |     #0 0x4c1873 in malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:88:3
[task 2018-07-21T12:51:18.143Z] 12:51:18     INFO - PID 11652 |     #1 0x4f2b7d in moz_xmalloc /builds/worker/workspace/build/src/memory/mozalloc/mozalloc.cpp:70:17
[task 2018-07-21T12:51:18.143Z] 12:51:18     INFO - PID 11652 |     #2 0x7f5bc3a8171a in operator new /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:136:12
[task 2018-07-21T12:51:18.144Z] 12:51:18     INFO - PID 11652 |     #3 0x7f5bc3a8171a in MakeUnique<mozilla::ChromiumCDMCallbackProxy, const RefPtr<mozilla::ChromiumCDMProxy> &, const nsCOMPtr<nsIEventTarget> &> /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/UniquePtr.h:680
[task 2018-07-21T12:51:18.145Z] 12:51:18     INFO - PID 11652 |     #4 0x7f5bc3a8171a in mozilla::ChromiumCDMProxy::Init(unsigned int, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&)::$_7::operator()() const::{lambda(RefPtr<mozilla::gmp::ChromiumCDMParent>)#1}::operator()(RefPtr<mozilla::gmp::ChromiumCDMParent>) const /builds/worker/workspace/build/src/dom/media/gmp/ChromiumCDMProxy.cpp:103
[task 2018-07-21T12:51:18.145Z] 12:51:18     INFO - PID 11652 |     #5 0x7f5bc3a80f69 in InvokeMethod<(lambda at /builds/worker/workspace/build/src/dom/media/gmp/ChromiumCDMProxy.cpp:101:9), void ((lambda at /builds/worker/workspace/build/src/dom/media/gmp/ChromiumCDMProxy.cpp:101:9)::*)(RefPtr<mozilla::gmp::ChromiumCDMParent>) const, RefPtr<mozilla::gmp::ChromiumCDMParent> > /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:529:12
[task 2018-07-21T12:51:18.147Z] 12:51:18     INFO - PID 11652 |     #6 0x7f5bc3a80f69 in InvokeCallbackMethod<false, (lambda at /builds/worker/workspace/build/src/dom/media/gmp/ChromiumCDMProxy.cpp:101:9), void ((lambda at /builds/worker/workspace/build/src/dom/media/gmp/ChromiumCDMProxy.cpp:101:9)::*)(RefPtr<mozilla::gmp::ChromiumCDMParent>) const, RefPtr<mozilla::gmp::ChromiumCDMParent>, RefPtr<mozilla::MozPromise<RefPtr<mozilla::gmp::ChromiumCDMParent>, mozilla::MediaResult, true>::Private> > /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:574
[task 2018-07-21T12:51:18.147Z] 12:51:18     INFO - PID 11652 |     #7 0x7f5bc3a80f69 in mozilla::MozPromise<RefPtr<mozilla::gmp::ChromiumCDMParent>, mozilla::MediaResult, true>::ThenValue<mozilla::ChromiumCDMProxy::Init(unsigned int, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&)::$_7::operator()() const::{lambda(RefPtr<mozilla::gmp::ChromiumCDMParent>)#1}, mozilla::ChromiumCDMProxy::Init(unsigned int, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&, nsTSubstring<char16_t> const&)::$_7::operator()() const::{lambda(mozilla::MediaResult)#1}>::DoResolveOrRejectInternal(mozilla::MozPromise<RefPtr<mozilla::gmp::ChromiumCDMParent>, mozilla::MediaResult, true>::ResolveOrRejectValue&) /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:774
[task 2018-07-21T12:51:18.149Z] 12:51:18     INFO - PID 11652 |     #8 0x7f5bc3a8242a in DoResolveOrReject /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:506:7
[task 2018-07-21T12:51:18.150Z] 12:51:18     INFO - PID 11652 |     #9 0x7f5bc3a8242a in mozilla::MozPromise<RefPtr<mozilla::gmp::ChromiumCDMParent>, mozilla::MediaResult, true>::ThenValueBase::ResolveOrRejectRunnable::Run() /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/MozPromise.h:411
[task 2018-07-21T12:51:18.150Z] 12:51:18     INFO - PID 11652 |     #10 0x7f5bbd4fd2cb in mozilla::EventTargetWrapper::Runner::Run() /builds/worker/workspace/build/src/xpcom/threads/AbstractThread.cpp:150:32
[task 2018-07-21T12:51:18.151Z] 12:51:18     INFO - PID 11652 |     #11 0x7f5bbd521e61 in nsThread::ProcessNextEvent(bool, bool*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:1166:14
[task 2018-07-21T12:51:18.152Z] 12:51:18     INFO - PID 11652 |     #12 0x7f5bbd5285d8 in NS_ProcessNextEvent(nsIThread*, bool) /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp:519:10
[task 2018-07-21T12:51:18.153Z] 12:51:18     INFO - PID 11652 |     #13 0x7f5bbe41fa06 in mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) /builds/worker/workspace/build/src/ipc/glue/MessagePump.cpp:334:20
[task 2018-07-21T12:51:18.153Z] 12:51:18     INFO - PID 11652 |     #14 0x7f5bbe3788bc in RunInternal /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:325:10
[task 2018-07-21T12:51:18.156Z] 12:51:18     INFO - PID 11652 |     #15 0x7f5bbe3788bc in RunHandler /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:318
[task 2018-07-21T12:51:18.156Z] 12:51:18     INFO - PID 11652 |     #16 0x7f5bbe3788bc in MessageLoop::Run() /builds/worker/workspace/build/src/ipc/chromium/src/base/message_loop.cc:298
[task 2018-07-21T12:51:18.157Z] 12:51:18     INFO - PID 11652 |     #17 0x7f5bbd51c5d3 in nsThread::ThreadFunc(void*) /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:525:11
[task 2018-07-21T12:51:18.157Z] 12:51:18     INFO - PID 11652 |     #18 0x7f5bd96d6dc8 in _pt_root /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:201:5
[task 2018-07-21T12:51:18.157Z] 12:51:18     INFO - PID 11652 |     #19 0x7f5bdd9f06b9 in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x76b9)
[task 2018-07-21T12:51:18.157Z] 12:51:18     INFO - PID 11652 | -----------------------------------------------------
[task 2018-07-21T12:51:18.158Z] 12:51:18     INFO - PID 11652 | Suppressions used:
[task 2018-07-21T12:51:18.158Z] 12:51:18     INFO - PID 11652 |   count      bytes template
[task 2018-07-21T12:51:18.158Z] 12:51:18     INFO - PID 11652 |     633      20192 nsComponentManagerImpl
[task 2018-07-21T12:51:18.159Z] 12:51:18     INFO - PID 11652 |       5        960 mozJSComponentLoader::LoadModule
[task 2018-07-21T12:51:18.159Z] 12:51:18     INFO - PID 11652 |     611      17713 libfontconfig.so
[task 2018-07-21T12:51:18.159Z] 12:51:18     INFO - PID 11652 |       1         29 libglib-2.0.so
[task 2018-07-21T12:51:18.159Z] 12:51:18     INFO - PID 11652 | -----------------------------------------------------
[task 2018-07-21T12:51:19.015Z] 12:51:19     INFO - PID 11652 | -----------------------------------------------------
[task 2018-07-21T12:51:19.015Z] 12:51:19     INFO - PID 11652 | Suppressions used:
[task 2018-07-21T12:51:19.015Z] 12:51:19     INFO - PID 11652 |   count      bytes template
[task 2018-07-21T12:51:19.016Z] 12:51:19     INFO - PID 11652 |     639      20384 nsComponentManagerImpl
[task 2018-07-21T12:51:19.016Z] 12:51:19     INFO - PID 11652 |      40       7680 mozJSComponentLoader::LoadModule
[task 2018-07-21T12:51:19.016Z] 12:51:19     INFO - PID 11652 |     611      17509 libfontconfig.so
[task 2018-07-21T12:51:19.017Z] 12:51:19     INFO - PID 11652 |      30       1320 _PR_Getfd
[task 2018-07-21T12:51:19.017Z] 12:51:19     INFO - PID 11652 |       1         29 libglib-2.0.so
[task 2018-07-21T12:51:19.018Z] 12:51:19     INFO - PID 11652 | -----------------------------------------------------
[task 2018-07-21T12:51:19.187Z] 12:51:19     INFO - Browser exited with return code 0
[task 2018-07-21T12:51:19.188Z] 12:51:19     INFO - ERROR | LeakSanitizer | SUMMARY: AddressSanitizer: 424 byte(s) leaked in 6 allocation(s).
[task 2018-07-21T12:51:19.188Z] 12:51:19     INFO - LeakSanitizer | To show the addresses of leaked objects add report_objects=1 to LSAN_OPTIONS
[task 2018-07-21T12:51:19.189Z] 12:51:19     INFO - This can be done in testing/mozbase/mozrunner/mozrunner/utils.py
[task 2018-07-21T12:51:19.189Z] 12:51:19     INFO - TEST-FAIL | LeakSanitizer | leak at Alloc, nsTSubstring, nsTSubstring, nsTSubstring
[task 2018-07-21T12:51:19.190Z] 12:51:19     INFO - INFO | LeakSanitizer | Frame Alloc matched a expected leak
[task 2018-07-21T12:51:19.190Z] 12:51:19     INFO - TEST-UNEXPECTED-FAIL | LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys
[task 2018-07-21T12:51:19.191Z] 12:51:19     INFO - TEST-FAIL | LeakSanitizer | leak at MakeUnique, mozilla::ChromiumCDMProxy::Init, InvokeMethod, InvokeCallbackMethod
[task 2018-07-21T12:51:19.191Z] 12:51:19     INFO - INFO | LeakSanitizer | Frame MakeUnique matched a expected leak
This is perma-failing beta simulations since Linux asan wpt got added last week.

Matthew, can you take a look at this, please?
Flags: needinfo?(kinetik)
I might not be the best person to look at this, I'm not super familiar with this code.  I can take it if there's nobody free or more suitable, though!  ni? Nils to check
Flags: needinfo?(kinetik) → needinfo?(drno)
Nils, could you have a look please? Thanks
Bryce could this be related to the Widevine update? Could you please have a look?
Flags: needinfo?(drno) → needinfo?(bvandyk)
Will do. Looks like this was an issue 4~ days before we switched over the CDM in bug 1475260, so it may have been a preexisting problem that has been highlighted by the changes mentioned in comment #1.
Flags: needinfo?(bvandyk)
This looks to be an intermittent caused by us not having an exclusion for one of the leaks found in these tests. When bug 1354232 landed it also landed a number of exclusions for related cases, but we're missing one for this case. We could just add a further exclusion to those already in place.

However, the ideal solution would be to not have leaks at all, rather than exclusions, assuming these aren't false positives. I don't know if the interaction with the CDM is special in some way that could result in false positives. I'll see if I can get some knowledge from someone with more lsan experience.

This does appear to be a preexisting problem as per comment #6, so these are at least not new leaks being introduced by recent changes.
Assignee: nobody → bvandyk
Summary: Perma-failing LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys when Gecko 63 merges to Beta on 2018-08-27 → Intermittent LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys when Gecko 63 merges to Beta on 2018-08-27
Looks like I'd misunderstood the context of this failure. It's perma-failing for our beta simulation, which I'm afraid I know little about. Reexamining the failure from the log:

> LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys 

and comparing to our allowed leaks:

> lsan-allowed: [Alloc, MakeUnique, Malloc, NewPage, Realloc, mozilla::EMEDecryptor::EMEDecryptor, mozilla::SchedulerGroup::CreateEventTargetFor, mozilla::dom::MediaKeys::CreateCDMProxy, mozilla::dom::nsIContentChild::GetConstructedEventTarget]

It looks like `CreateCDMProxy` is not fully qualified in our failure for the simulation. Checking on central try runs I see a similar stack passing because it matches our allowed list:

> LeakSanitizer | leak at mozilla::dom::MediaKeys::CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys

^ This is fine and these tests are marked as failures.

So this issue is due to `CreateCDMProxy` not being fully qualified in our beta sim try runs. I don't know what causes that, but wonder if there's a straight forward explanation. Ryan, I note you created the beta patches for these try runs, do you know of anything that would cause the difference in logging for the beta simulations?
Flags: needinfo?(ryanvm)
Summary: Intermittent LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys when Gecko 63 merges to Beta on 2018-08-27 → Perma-failing LeakSanitizer | leak at CreateCDMProxy, mozilla::dom::MediaKeys::Init, mozilla::dom::MediaKeySystemAccess::CreateMediaKeys, createMediaKeys when Gecko 63 merges to Beta on 2018-08-27
The beta simulations are pushes on the Try server and these don't have access to secrets. The maple repository is also affected by the issue: https://treeherder.mozilla.org/#/jobs?repo=maple&revision=efcba08f4a354ea0b023f0d31c9564360bed1d03&filter-resultStatus=testfailed&filter-resultStatus=busted&filter-resultStatus=exception&filter-resultStatus=retry&filter-resultStatus=usercancel&filter-resultStatus=runnable&selectedJob=195149316 (one recent passing job, the others failed).

Is this difference between a stub CDM and real CDM?
Flags: needinfo?(ryanvm)
Forgive my lack of familiarity, but could you elaborate on the kinds of information which are secrets and not available to these builds?

These tests use the clearkey key system, so we will use the clearkey implementation in the browser rather than using the widevine CDM we fetch. The clearkey impl we have will still create CDM machinery and invoke the functions we see in the failing stack. This should happen irrespective of environment, and I wouldn't expect differences in clearkey behaviour between environments, nor anything leading to a different stack (at least at the frames we're looking at here).

It looks like this failure is an issue of the allowed list not working as expected. It would be good to fix the leak, but the leak seems to happen elsewhere and gets allowed.

There's only one CreateCDMProxy[0] in our C++ code right now. In these failing logs its not fully qualified. My understanding for the lsan-allowed list is that we perform a string match on the entries and all the stack frames in a leak. Since the unqualified function doesn't match in this case, we fail.

Hopefully these questions aren't too silly, but is there something that could lead to these builds having that function truncated in the lsan log?
- Different symbolizer behaviour?
- Log trimming?

[0]: https://searchfox.org/mozilla-central/rev/3fa761ade83ed0b8ab463acb057c2cf0b104689e/dom/media/eme/MediaKeys.h#153
Andrew, can you provide information on Bryce's questions about the lsan log? Thank you in advance.
Flags: needinfo?(continuation)
That's pretty strange. We might use a different version of Clang on beta, but I don't know why that would cause the qualifiers to be dropped. If the stacks look they are basically the same thing, I'd just add an additional thing to the allowed list that covers the way they look on beta, if that works. Just indicate why you are doing it.
Flags: needinfo?(continuation)
CreateCDMProxy in its fully qualified form is already an allowed leak, but per
bug 1477492 certain kinds of builds appear to log the unqualified function name
in leak sanitizer logs. Since the allowed list works on string matching, this
unqualified version is then not matched and tests are marked as failures. As the
source of this different behaviour is not clear, this patch makes the
unqualified function name also allowed.
Comment on attachment 9003461 [details]
Bug 1477492 - Add 'CreateCDMProxy' to list of allowed leaks for encrypted-media wpt.

James Graham [:jgraham] has approved the revision.
Attachment #9003461 - Flags: review+
Confirming via try run that the patch above mitigates the issue. Comparing before[0] and after[1] shows the specific problem noted in this bug is mitigated, though the test still fails due to other problems.

[0]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=23937fc4d8a0dd57e1b4918c0222202bf94a2669&selectedJob=195530365
[1]: https://treeherder.mozilla.org/#/jobs?repo=try&revision=fb3aee27250a0f5be040875813666dd45bffe565&selectedJob=195530095
Pushed by bvandyk@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/68ddeae44caa
Add 'CreateCDMProxy' to list of allowed leaks for encrypted-media wpt. r=jgraham
Priority: -- → P2
https://hg.mozilla.org/mozilla-central/rev/68ddeae44caa
Status: NEW → RESOLVED
Closed: Last year
Resolution: --- → FIXED
Target Milestone: --- → mozilla63
See Also: → 1486389
Component: DOM → DOM: Core & HTML
You need to log in before you can comment on or make changes to this bug.