Bug 1551615 Comment 67 Edit History

Note: The actual edited comment in the bug view page will always show the original commenter’s name and original timestamp.

Original comment by

Andreea Pavel [:apavel]

on 2020-05-17 12:52:52 PDT

There are 74 total failures with a spike starting with the 15th of may, all on linux1804-64-asan opt

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302624975&repo=autoland&lineNumber=3508

> [task 2020-05-17T07:50:55.470Z] 07:50:55     INFO - TEST-START | /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html
[task 2020-05-17T07:50:55.499Z] 07:50:55     INFO - Closing window 47
[task 2020-05-17T07:50:55.804Z] 07:50:55     INFO - PID 5116 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-05-17T07:50:55.825Z] 07:50:55     INFO - PID 5116 | AddressSanitizer:DEADLYSIGNAL
[task 2020-05-17T07:50:55.828Z] 07:50:55     INFO - PID 5116 | =================================================================
[task 2020-05-17T07:50:55.830Z] 07:50:55    ERROR - PID 5116 | ==6550==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fbeb89ee901 bp 0x7ffc9652fb50 sp 0x7ffc9652f980 T0)
[task 2020-05-17T07:50:55.830Z] 07:50:55     INFO - PID 5116 | ==6550==The signal is caused by a WRITE memory access.
[task 2020-05-17T07:50:55.830Z] 07:50:55     INFO - PID 5116 | ==6550==Hint: address points to the zero page.
[task 2020-05-17T07:50:56.796Z] 07:50:56     INFO - PID 5116 |     #0 0x7fbeb89ee901 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9
[task 2020-05-17T07:50:56.797Z] 07:50:56     INFO - PID 5116 |     #1 0x7fbeb89ed3f6 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-05-17T07:50:56.801Z] 07:50:56     INFO - PID 5116 |     #2 0x7fbeb8a133b2 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-05-17T07:50:56.821Z] 07:50:56     INFO - PID 5116 |     #3 0x7fbebd77b7b7 in Release /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.h:25:3
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #4 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:50:40
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #5 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:381:36
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #6 0x7fbebd77b7b7 in ~RefPtr /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:81:7
[task 2020-05-17T07:50:56.823Z] 07:50:56     INFO - PID 5116 |     #7 0x7fbebd77b7b7 in ~nsBaseHashtableET /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:72:32
[task 2020-05-17T07:50:56.825Z] 07:50:56     INFO - PID 5116 |     #8 0x7fbebd77b7b7 in nsTHashtable<nsBaseHashtableET<nsUint64HashKey, RefPtr<mozilla::gmp::GMPContentParent> > >::s_ClearEntry(PLDHashTable*, PLDHashEntryHdr*) /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:512:37
[task 2020-05-17T07:50:56.842Z] 07:50:56     INFO - PID 5116 |     #9 0x7fbeb77a9866 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-05-17T07:50:56.843Z] 07:50:56     INFO - PID 5116 |     #10 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-05-17T07:50:56.845Z] 07:50:56     INFO - PID 5116 |     #11 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-05-17T07:50:56.847Z] 07:50:56     INFO - PID 5116 |     #12 0x7fbeb77a9866 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-05-17T07:50:56.848Z] 07:50:56     INFO - PID 5116 |     #13 0x7fbebd74e64e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-05-17T07:50:56.849Z] 07:50:56     INFO - PID 5116 |     #14 0x7fbebd74e64e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.850Z] 07:50:56     INFO - PID 5116 |     #15 0x7fbebd74e64e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.854Z] 07:50:56     INFO - PID 5116 |     #16 0x7fbebd74d3bd in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-05-17T07:50:56.855Z] 07:50:56     INFO - PID 5116 |     #17 0x7fbebd74d3bd in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-05-17T07:50:56.856Z] 07:50:56     INFO - PID 5116 |     #18 0x7fbebd74d3bd in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-05-17T07:50:56.857Z] 07:50:56     INFO - PID 5116 |     #19 0x7fbebd74d3bd in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:357:21
[task 2020-05-17T07:50:56.871Z] 07:50:56     INFO - PID 5116 |     #20 0x7fbebd74d4cc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-05-17T07:50:56.873Z] 07:50:56     INFO - PID 5116 |     #21 0x7fbeb77c7bbe in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-05-17T07:50:56.874Z] 07:50:56     INFO - PID 5116 |     #22 0x7fbeb77ceb91 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:288:19
[task 2020-05-17T07:50:56.874Z] 07:50:56     INFO - PID 5116 |     #23 0x7fbeb796a1e3 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:638:24
[task 2020-05-17T07:50:56.875Z] 07:50:56     INFO - PID 5116 |     #24 0x7fbec254a96c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-05-17T07:50:56.876Z] 07:50:56     INFO - PID 5116 |     #25 0x7fbeb8a25b9e in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-05-17T07:50:56.876Z] 07:50:56     INFO - PID 5116 |     #26 0x7fbec254ba25 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-05-17T07:50:56.890Z] 07:50:56     INFO - PID 5116 |     #27 0x55daf4fcc16d in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-05-17T07:50:56.890Z] 07:50:56     INFO - PID 5116 |     #28 0x55daf4fcc5aa in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 |     #29 0x7fbed78c4b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 |     #30 0x55daf4f20cb5 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x55cb5)
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 | AddressSanitizer can not provide additional info.
[task 2020-05-17T07:50:57.008Z] 07:50:57     INFO - PID 5116 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-05-17T07:50:57.009Z] 07:50:57     INFO - PID 5116 | ==6550==ABORTING
[task 2020-05-17T07:50:57.087Z] 07:50:57     INFO - .
[task 2020-05-17T07:50:57.087Z] 07:50:57     INFO - TEST-OK | /encoding-detection/zh-Big5-late.tentative.html | took 3082ms

Looked in last 15 failure logs and this fails in:
- html/cross-origin-embedder-policy/blob.https.html
- encrypted-media/clearkey-mp4-waiting-for-a-key.https.html
- encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html	
- eventsource/format-field-unknown.htm

Revision 1 by

Andreea Pavel [:apavel]

on 2020-05-17 13:14:56 PDT

There are 74 total failures in the last 7 days all on linux1804-64-asan opt, with failure count going up on the 15th:

https://treeherder.mozilla.org/intermittent-failures.html#/bugdetails?startday=2020-05-10&endday=2020-05-17&tree=trunk&bug=1551615

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=302624975&repo=autoland&lineNumber=3508

> [task 2020-05-17T07:50:55.470Z] 07:50:55     INFO - TEST-START | /encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html
[task 2020-05-17T07:50:55.499Z] 07:50:55     INFO - Closing window 47
[task 2020-05-17T07:50:55.804Z] 07:50:55     INFO - PID 5116 | ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2020-05-17T07:50:55.825Z] 07:50:55     INFO - PID 5116 | AddressSanitizer:DEADLYSIGNAL
[task 2020-05-17T07:50:55.828Z] 07:50:55     INFO - PID 5116 | =================================================================
[task 2020-05-17T07:50:55.830Z] 07:50:55    ERROR - PID 5116 | ==6550==ERROR: AddressSanitizer: SEGV on unknown address 0x000000000001 (pc 0x7fbeb89ee901 bp 0x7ffc9652fb50 sp 0x7ffc9652f980 T0)
[task 2020-05-17T07:50:55.830Z] 07:50:55     INFO - PID 5116 | ==6550==The signal is caused by a WRITE memory access.
[task 2020-05-17T07:50:55.830Z] 07:50:55     INFO - PID 5116 | ==6550==Hint: address points to the zero page.
[task 2020-05-17T07:50:56.796Z] 07:50:56     INFO - PID 5116 |     #0 0x7fbeb89ee901 in mozilla::ipc::MessageChannel::Clear() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9
[task 2020-05-17T07:50:56.797Z] 07:50:56     INFO - PID 5116 |     #1 0x7fbeb89ed3f6 in mozilla::ipc::MessageChannel::~MessageChannel() /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:633:3
[task 2020-05-17T07:50:56.801Z] 07:50:56     INFO - PID 5116 |     #2 0x7fbeb8a133b2 in mozilla::ipc::IToplevelProtocol::~IToplevelProtocol() /builds/worker/workspace/obj-build/dist/include/mozilla/ipc/ProtocolUtils.h:391:32
[task 2020-05-17T07:50:56.821Z] 07:50:56     INFO - PID 5116 |     #3 0x7fbebd77b7b7 in Release /builds/worker/checkouts/gecko/dom/media/gmp/GMPContentParent.h:25:3
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #4 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:50:40
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #5 0x7fbebd77b7b7 in Release /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:381:36
[task 2020-05-17T07:50:56.822Z] 07:50:56     INFO - PID 5116 |     #6 0x7fbebd77b7b7 in ~RefPtr /builds/worker/workspace/obj-build/dist/include/mozilla/RefPtr.h:81:7
[task 2020-05-17T07:50:56.823Z] 07:50:56     INFO - PID 5116 |     #7 0x7fbebd77b7b7 in ~nsBaseHashtableET /builds/worker/workspace/obj-build/dist/include/nsBaseHashtable.h:72:32
[task 2020-05-17T07:50:56.825Z] 07:50:56     INFO - PID 5116 |     #8 0x7fbebd77b7b7 in nsTHashtable<nsBaseHashtableET<nsUint64HashKey, RefPtr<mozilla::gmp::GMPContentParent> > >::s_ClearEntry(PLDHashTable*, PLDHashEntryHdr*) /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:512:37
[task 2020-05-17T07:50:56.842Z] 07:50:56     INFO - PID 5116 |     #9 0x7fbeb77a9866 in operator() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:292:7
[task 2020-05-17T07:50:56.843Z] 07:50:56     INFO - PID 5116 |     #10 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:357:9
[task 2020-05-17T07:50:56.845Z] 07:50:56     INFO - PID 5116 |     #11 0x7fbeb77a9866 in ForEachSlot<(lambda at /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:51)> /builds/worker/workspace/obj-build/dist/include/PLDHashTable.h:347:7
[task 2020-05-17T07:50:56.847Z] 07:50:56     INFO - PID 5116 |     #12 0x7fbeb77a9866 in PLDHashTable::~PLDHashTable() /builds/worker/checkouts/gecko/xpcom/ds/PLDHashTable.cpp:290:15
[task 2020-05-17T07:50:56.848Z] 07:50:56     INFO - PID 5116 |     #13 0x7fbebd74e64e in ~nsTHashtable /builds/worker/workspace/obj-build/dist/include/nsTHashtable.h:170:27
[task 2020-05-17T07:50:56.849Z] 07:50:56     INFO - PID 5116 |     #14 0x7fbebd74e64e in ~GMPServiceChild /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.850Z] 07:50:56     INFO - PID 5116 |     #15 0x7fbebd74e64e in mozilla::gmp::GMPServiceChild::~GMPServiceChild() /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:426:35
[task 2020-05-17T07:50:56.854Z] 07:50:56     INFO - PID 5116 |     #16 0x7fbebd74d3bd in operator() /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:460:5
[task 2020-05-17T07:50:56.855Z] 07:50:56     INFO - PID 5116 |     #17 0x7fbebd74d3bd in reset /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:302:7
[task 2020-05-17T07:50:56.856Z] 07:50:56     INFO - PID 5116 |     #18 0x7fbebd74d3bd in operator= /builds/worker/workspace/obj-build/dist/include/mozilla/UniquePtr.h:275:5
[task 2020-05-17T07:50:56.857Z] 07:50:56     INFO - PID 5116 |     #19 0x7fbebd74d3bd in mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp:357:21
[task 2020-05-17T07:50:56.871Z] 07:50:56     INFO - PID 5116 |     #20 0x7fbebd74d4cc in non-virtual thunk to mozilla::gmp::GeckoMediaPluginServiceChild::Observe(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/dom/media/gmp/GMPServiceChild.cpp
[task 2020-05-17T07:50:56.873Z] 07:50:56     INFO - PID 5116 |     #21 0x7fbeb77c7bbe in nsObserverList::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverList.cpp:65:19
[task 2020-05-17T07:50:56.874Z] 07:50:56     INFO - PID 5116 |     #22 0x7fbeb77ceb91 in nsObserverService::NotifyObservers(nsISupports*, char const*, char16_t const*) /builds/worker/checkouts/gecko/xpcom/ds/nsObserverService.cpp:288:19
[task 2020-05-17T07:50:56.874Z] 07:50:56     INFO - PID 5116 |     #23 0x7fbeb796a1e3 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/checkouts/gecko/xpcom/build/XPCOMInit.cpp:638:24
[task 2020-05-17T07:50:56.875Z] 07:50:56     INFO - PID 5116 |     #24 0x7fbec254a96c in XRE_TermEmbedding() /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:222:3
[task 2020-05-17T07:50:56.876Z] 07:50:56     INFO - PID 5116 |     #25 0x7fbeb8a25b9e in mozilla::ipc::ScopedXREEmbed::Stop() /builds/worker/checkouts/gecko/ipc/glue/ScopedXREEmbed.cpp:90:5
[task 2020-05-17T07:50:56.876Z] 07:50:56     INFO - PID 5116 |     #26 0x7fbec254ba25 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/checkouts/gecko/toolkit/xre/nsEmbedFunctions.cpp:744:16
[task 2020-05-17T07:50:56.890Z] 07:50:56     INFO - PID 5116 |     #27 0x55daf4fcc16d in content_process_main(mozilla::Bootstrap*, int, char**) /builds/worker/checkouts/gecko/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2020-05-17T07:50:56.890Z] 07:50:56     INFO - PID 5116 |     #28 0x55daf4fcc5aa in main /builds/worker/checkouts/gecko/browser/app/nsBrowserApp.cpp:303:18
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 |     #29 0x7fbed78c4b96 in __libc_start_main /build/glibc-OTsEL5/glibc-2.27/csu/../csu/libc-start.c:310
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 |     #30 0x55daf4f20cb5 in _start (/builds/worker/workspace/build/application/firefox/firefox+0x55cb5)
[task 2020-05-17T07:50:57.003Z] 07:50:57     INFO - PID 5116 | AddressSanitizer can not provide additional info.
[task 2020-05-17T07:50:57.008Z] 07:50:57     INFO - PID 5116 | SUMMARY: AddressSanitizer: SEGV /builds/worker/checkouts/gecko/ipc/glue/MessageChannel.cpp:749:9 in mozilla::ipc::MessageChannel::Clear()
[task 2020-05-17T07:50:57.009Z] 07:50:57     INFO - PID 5116 | ==6550==ABORTING
[task 2020-05-17T07:50:57.087Z] 07:50:57     INFO - .
[task 2020-05-17T07:50:57.087Z] 07:50:57     INFO - TEST-OK | /encoding-detection/zh-Big5-late.tentative.html | took 3082ms

Looked in last 15 failure logs and this fails in:
- html/cross-origin-embedder-policy/blob.https.html
- encrypted-media/clearkey-mp4-waiting-for-a-key.https.html
- encrypted-media/clearkey-mp4-playback-temporary-multikey-sequential.https.html	
- eventsource/format-field-unknown.htm

Back to Bug 1551615 Comment 67