Closed Bug 1445822 Opened 6 years ago Closed 5 years ago

Intermittent Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/js/src/vm/ProfilingStack.cpp:22

Categories

(Core :: Gecko Profiler, defect, P5)

Product:
Core
Component:
Gecko Profiler
Type:
defect

Tracking

()

Status:
RESOLVED FIXED
Milestone:
mozilla72
Tracking Flags:
Tracking Status
firefox-esr68 --- fixed
firefox70 --- wontfix
firefox71 --- wontfix
firefox72 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: mozbugz)

References

Details

(Keywords: assertion, intermittent-failure, Whiteboard: [comment64][stockwell unknown])

Crash Data

Attachments

(2 files, 1 obsolete file)

Bug 1445822 - Make ProfilingStack ref-counted, owned by both profiler and thread - r?mstange
47 bytes, text/x-phabricator-request
Details | Review
Bug 1445822 - Dump stack and crash when ProfilingStack is not empty on destruction - r?mstange
47 bytes, text/x-phabricator-request
Details | Review 
Filed by: apavel [at] mozilla.com

https://treeherder.mozilla.org/logviewer.html#?job_id=168070277&repo=autoland

https://queue.taskcluster.net/v1/task/TLF4TLRaQDeLHPVj2UCMFg/runs/0/artifacts/public/logs/live_backing.log

https://hg.mozilla.org/mozilla-central/raw-file/tip/layout/tools/reftest/reftest-analyzer.xhtml#logurl=https://queue.taskcluster.net/v1/task/TLF4TLRaQDeLHPVj2UCMFg/runs/0/artifacts/public/logs/live_backing.log&only_show_unexpected=1

[task 2018-03-14T23:06:16.697Z] 23:06:16     INFO - [Parent 1114, Main Thread] WARNING: '!mMainThread', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 517
[task 2018-03-14T23:06:16.698Z] 23:06:16     INFO - --DOMWINDOW == 9 (0x7f20b5cb4400) [pid = 1114] [serial = 13] [outer = (nil)] [url = chrome://extensions/content/dummy.xul]
[task 2018-03-14T23:06:16.698Z] 23:06:16     INFO - --DOMWINDOW == 8 (0x7f20b5cb2000) [pid = 1114] [serial = 10] [outer = (nil)] [url = chrome://extensions/content/dummy.xul]
[task 2018-03-14T23:06:16.699Z] 23:06:16     INFO - --DOMWINDOW == 7 (0x7f20d438d800) [pid = 1114] [serial = 18] [outer = (nil)] [url = about:blank]
[task 2018-03-14T23:06:16.700Z] 23:06:16     INFO - --DOMWINDOW == 6 (0x7f20d41cd000) [pid = 1114] [serial = 17] [outer = (nil)] [url = about:blank]
[task 2018-03-14T23:06:16.701Z] 23:06:16     INFO - --DOMWINDOW == 5 (0x7f20d4386800) [pid = 1114] [serial = 1] [outer = (nil)] [url = resource://gre-resources/hiddenWindow.html]
[task 2018-03-14T23:06:16.702Z] 23:06:16     INFO - --DOMWINDOW == 4 (0x7f20d415d400) [pid = 1114] [serial = 3] [outer = (nil)] [url = chrome://browser/content/browser.xul]
[task 2018-03-14T23:06:16.702Z] 23:06:16     INFO - --DOMWINDOW == 3 (0x7f20bd655800) [pid = 1114] [serial = 20] [outer = (nil)] [url = about:blank]
[task 2018-03-14T23:06:16.704Z] 23:06:16     INFO - --DOMWINDOW == 2 (0x7f20bd655000) [pid = 1114] [serial = 19] [outer = (nil)] [url = chrome://reftest/content/reftest.xul]
[task 2018-03-14T23:06:16.707Z] 23:06:16     INFO - --DOMWINDOW == 1 (0x7f20d432ac00) [pid = 1114] [serial = 5] [outer = (nil)] [url = resource://gre-resources/hiddenWindow.html]
[task 2018-03-14T23:06:16.709Z] 23:06:16     INFO - --DOMWINDOW == 0 (0x7f20b5cb7800) [pid = 1114] [serial = 16] [outer = (nil)] [url = moz-extension://4a27f701-ef91-4278-8a1b-0f0d7c6a5b04/_generated_background_page.html]
[task 2018-03-14T23:06:16.770Z] 23:06:16     INFO - Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/obj-firefox/dist/include/js/ProfilingStack.h:308
[task 2018-03-14T23:06:16.770Z] 23:06:16     INFO - #01: CorePS::~CorePS [memory/mozalloc/mozalloc.h:180]
[task 2018-03-14T23:06:16.771Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.772Z] 23:06:16     INFO - #02: profiler_shutdown [memory/mozalloc/mozalloc.h:180]
[task 2018-03-14T23:06:16.773Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.773Z] 23:06:16     INFO - #03: XREMain::XRE_main [tools/profiler/public/GeckoProfiler.h:599]
[task 2018-03-14T23:06:16.774Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.775Z] 23:06:16     INFO - #04: XRE_main [toolkit/xre/nsAppRunner.cpp:4961]
[task 2018-03-14T23:06:16.775Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.776Z] 23:06:16     INFO - #05: do_main [browser/app/nsBrowserApp.cpp:231]
[task 2018-03-14T23:06:16.776Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.777Z] 23:06:16     INFO - #06: main [browser/app/nsBrowserApp.cpp:306]
[task 2018-03-14T23:06:16.777Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.779Z] 23:06:16     INFO - #07: libc.so.6 + 0x20830
[task 2018-03-14T23:06:16.779Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.780Z] 23:06:16     INFO - #08: _start
[task 2018-03-14T23:06:16.781Z] 23:06:16     INFO - 
[task 2018-03-14T23:06:16.818Z] 23:06:16    ERROR - TEST-UNEXPECTED-FAIL | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/bugs/1432541.html | application terminated with exit code -11
[task 2018-03-14T23:06:16.826Z] 23:06:16     INFO - REFTEST INFO | Process mode: e10s
[task 2018-03-14T23:06:16.828Z] 23:06:16  WARNING - leakcheck | refcount logging is off, so leaks can't be detected!
[task 2018-03-14T23:06:16.829Z] 23:06:16     INFO - REFTEST INFO | Result summary:
[task 2018-03-14T23:06:16.830Z] 23:06:16     INFO - REFTEST INFO | Successful: 1914 (1914 pass, 0 load only)
[task 2018-03-14T23:06:16.831Z] 23:06:16     INFO - REFTEST INFO | Unexpected: 0 (0 unexpected fail, 0 unexpected pass, 0 unexpected asserts, 0 failed load, 0 exception)
[task 2018-03-14T23:06:16.832Z] 23:06:16     INFO - REFTEST INFO | Known problems: 113 (42 known fail, 0 known asserts, 20 random, 51 skipped, 0 slow)
[task 2018-03-14T23:06:16.833Z] 23:06:16     INFO - REFTEST SUITE-END | Shutdown
[task 2018-03-14T23:06:17.627Z] 23:06:17    ERROR - Return code: 245
[task 2018-03-14T23:06:17.629Z] 23:06:17     INFO - TinderboxPrint: reftest-reftest<br/>5979/0/53
[task 2018-03-14T23:06:17.629Z] 23:06:17    ERROR - # TBPL FAILURE #
[task 2018-03-14T23:06:17.630Z] 23:06:17  WARNING - setting return code to 2
[task 2018-03-14T23:06:17.631Z] 23:06:17    ERROR - The reftest suite: reftest ran with return status: FAILURE
[task 2018-03-14T23:06:17.631Z] 23:06:17     INFO - Running post-action listener: _package_coverage_data
[task 2018-03-14T23:06:17.632Z] 23:06:17     INFO - Running post-action listener: _resource_record_post_action
[task 2018-03-14T23:06:17.632Z] 23:06:17     INFO - [mozharness: 2018-03-14 23:06:17.631733Z] Finished run-tests step (success)
[task 2018-03-14T23:06:17.633Z] 23:06:17     INFO - Running post-run listener: _resource_record_post_run
[task 2018-03-14T23:06:17.699Z] 23:06:17     INFO - Validating Perfherder data against /builds/worker/workspace/mozharness/external_tools/performance-artifact-schema.json
[task 2018-03-14T23:06:17.705Z] 23:06:17     INFO - PERFHERDER_DATA: {"framework": {"name": "job_resource_usage"}, "suites": [{"subtests": [{"name": "cpu_percent", "value": 72.44530321046375}, {"name": "io_write_bytes", "value": 754892800}, {"name": "io.read_bytes", "value": 46682112}, {"name": "io_write_time", "value": 79308}, {"name": "io_read_time", "value": 4088}], "extraOptions": ["e10s", "taskcluster-m3.large"], "name": "reftest.reftest.8.overall"}, {"subtests": [{"name": "time", "value": 18.83381986618042}, {"name": "cpu_percent", "value": 50.24444444444444}], "name": "reftest.reftest.8.install"}, {"subtests": [{"name": "time", "value": 0.0019259452819824219}], "name": "reftest.reftest.8.stage-files"}, {"subtests": [{"name": "time", "value": 824.1506719589233}, {"name": "cpu_percent", "value": 72.9557177615572}], "name": "reftest.reftest.8.run-tests"}]}
[task 2018-03-14T23:06:17.706Z] 23:06:17     INFO - Total resource usage - Wall time: 843s; CPU: 73.0%; Read bytes: 46682112; Write bytes: 754892800; Read time: 4088; Write time: 79308
[task 2018-03-14T23:06:17.706Z] 23:06:17     INFO - TinderboxPrint: CPU usage<br/>73.0%
[task 2018-03-14T23:06:17.707Z] 23:06:17     INFO - TinderboxPrint: I/O read bytes / time<br/>46,682,112 / 4,088
[task 2018-03-14T23:06:17.707Z] 23:06:17     INFO - TinderboxPrint: I/O write bytes / time<br/>754,892,800 / 79,308
[task 2018-03-14T23:06:17.708Z] 23:06:17     INFO - TinderboxPrint: CPU idle<br/>441.8 (27.4%)
[task 2018-03-14T23:06:17.709Z] 23:06:17     INFO - TinderboxPrint: CPU system<br/>127.8 (7.9%)
[task 2018-03-14T23:06:17.710Z] 23:06:17     INFO - TinderboxPrint: CPU user<br/>1,038.9 (64.4%)
[task 2018-03-14T23:06:17.710Z] 23:06:17     INFO - TinderboxPrint: Swap in / out<br/>0 / 0
[task 2018-03-14T23:06:17.711Z] 23:06:17     INFO - install - Wall time: 19s; CPU: 50.0%; Read bytes: 0; Write bytes: 12931072; Read time: 0; Write time: 1772
[task 2018-03-14T23:06:17.711Z] 23:06:17     INFO - stage-files - Wall time: 0s; CPU: Can't collect data; Read bytes: 0; Write bytes: 0; Read time: 0; Write time: 0
[task 2018-03-14T23:06:17.718Z] 23:06:17     INFO - run-tests - Wall time: 824s; CPU: 73.0%; Read bytes: 46682112; Write bytes: 741961728; Read time: 4088; Write time: 77536
[task 2018-03-14T23:06:17.920Z] 23:06:17     INFO - Running post-run listener: _upload_blobber_files
[task 2018-03-14T23:06:17.922Z] 23:06:17  WARNING - Blob upload gear skipped. Missing cmdline options.
[task 2018-03-14T23:06:17.922Z] 23:06:17     INFO - Running post-run listener: copy_logs_to_upload_dir
[task 2018-03-14T23:06:17.922Z] 23:06:17     INFO - Copying logs to upload dir...
[task 2018-03-14T23:06:17.922Z] 23:06:17     INFO - mkdir: /builds/worker/workspace/build/upload/logs
[task 2018-03-14T23:06:17.934Z] 23:06:17     INFO - Copying logs to upload dir...
[task 2018-03-14T23:06:17.947Z] 23:06:17  WARNING - returning nonzero exit status 2
[task 2018-03-14T23:06:17.967Z] cleanup
Reclassifying under Gecko Profiler, based on the backtrace (this is in GeckoProfiler.h /  "profiler_shutdown")
Component: Layout → Gecko Profiler
We should try to catch this on every push/pop
[Updating summary, as the assertion has moved to a different file via bug 1436179]
Summary: Intermittent Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/obj-firefox/dist/include/js/ProfilingStack.h:308 → Intermittent Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/js/src/vm/ProfilingStack.cpp:22
https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup
Status: NEW → RESOLVED
Closed: 6 years ago
Resolution: --- → INCOMPLETE

https://wiki.mozilla.org/Bug_Triage#Intermittent_Test_Failure_Cleanup

Status: REOPENED → RESOLVED
Closed: 6 years ago → 5 years ago
Resolution: --- → INCOMPLETE

New occurrence:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=249598128&repo=autoland&lineNumber=5525

Status: RESOLVED → REOPENED
Resolution: INCOMPLETE → ---
Crash Signature: [@ ProfilingStack::~ProfilingStack()]

There are 26 total failures in the last 7 days on linux64-qr debug.

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=257608088&repo=mozilla-inbound&lineNumber=1619

[task 2019-07-21T23:24:50.094Z] 23:24:50 INFO - TEST-START | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_storage_tab.js
[task 2019-07-21T23:24:51.187Z] 23:24:51 INFO - mozcrash Saved minidump as /builds/worker/workspace/build/blobber_upload_dir/02a35dc3-c62b-7390-1a48-c23ff5e22d7b.dmp
[task 2019-07-21T23:24:51.190Z] 23:24:51 WARNING - PROCESS-CRASH | xpcshell-remote.ini:toolkit/components/extensions/test/xpcshell/test_ext_same_site_cookies.js | application crashed [@ ProfilingStack::~ProfilingStack()]
[task 2019-07-21T23:24:51.191Z] 23:24:51 INFO - Crash dump filename: /tmp/xpc-other-9seu4L/02a35dc3-c62b-7390-1a48-c23ff5e22d7b.dmp
[task 2019-07-21T23:24:51.191Z] 23:24:51 INFO - Operating system: Linux
[task 2019-07-21T23:24:51.192Z] 23:24:51 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2019-07-21T23:24:51.194Z] 23:24:51 INFO - CPU: amd64
[task 2019-07-21T23:24:51.194Z] 23:24:51 INFO - family 6 model 62 stepping 4
[task 2019-07-21T23:24:51.195Z] 23:24:51 INFO - 2 CPUs
[task 2019-07-21T23:24:51.195Z] 23:24:51 INFO - GPU: UNKNOWN
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - Crash address: 0x0
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - Process uptime: not available
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - Thread 0 (crashed)
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - 0 libxul.so!ProfilingStack::~ProfilingStack() [ProfilingStack.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 21 + 0x29]
[task 2019-07-21T23:24:51.199Z] 23:24:51 INFO - rax = 0x00007fc879759582 rdx = 0x0000000000000000
[task 2019-07-21T23:24:51.200Z] 23:24:51 INFO - rcx = 0x0000558cf5e54b08 rbx = 0x00007fc861d7c0b0
[task 2019-07-21T23:24:51.202Z] 23:24:51 INFO - rsi = 0x00007fc871126770 rdi = 0x00007fc871125540
[task 2019-07-21T23:24:51.202Z] 23:24:51 INFO - rbp = 0x00007ffdf9d834c0 rsp = 0x00007ffdf9d834c0
[task 2019-07-21T23:24:51.203Z] 23:24:51 INFO - r8 = 0x00007fc871126770 r9 = 0x00007fc86673f480
[task 2019-07-21T23:24:51.203Z] 23:24:51 INFO - r10 = 0x0000000000000000 r11 = 0x0000000000000000
[task 2019-07-21T23:24:51.204Z] 23:24:51 INFO - r12 = 0x00007ffdf9d835a8 r13 = 0x00007ffdf9d839e8
[task 2019-07-21T23:24:51.205Z] 23:24:51 INFO - r14 = 0x00007fc866577048 r15 = 0x00007fc86140a060
[task 2019-07-21T23:24:51.206Z] 23:24:51 INFO - rip = 0x00007fc876551583
[task 2019-07-21T23:24:51.207Z] 23:24:51 INFO - Found by: given as instruction pointer in context
[task 2019-07-21T23:24:51.208Z] 23:24:51 INFO - 1 libxul.so!mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >::reset(RegisteredThread*) [UniquePtr.h:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 323 + 0x8]
[task 2019-07-21T23:24:51.209Z] 23:24:51 INFO - rbx = 0x00007fc861d7c0b0 rbp = 0x00007ffdf9d834e0
[task 2019-07-21T23:24:51.210Z] 23:24:51 INFO - rsp = 0x00007ffdf9d834d0 r12 = 0x00007ffdf9d835a8
[task 2019-07-21T23:24:51.210Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e8 r14 = 0x00007fc866577048
[task 2019-07-21T23:24:51.218Z] 23:24:51 INFO - r15 = 0x00007fc86140a060 rip = 0x00007fc876186b33
[task 2019-07-21T23:24:51.218Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.219Z] 23:24:51 INFO - 2 libxul.so!mozilla::Vector<mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >, 0ul, mozilla::MallocAllocPolicy>::~Vector() [Vector.h:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 908 + 0x10]
[task 2019-07-21T23:24:51.221Z] 23:24:51 INFO - rbx = 0x00007fc86140a048 rbp = 0x00007ffdf9d83530
[task 2019-07-21T23:24:51.222Z] 23:24:51 INFO - rsp = 0x00007ffdf9d834f0 r12 = 0x00007ffdf9d835a8
[task 2019-07-21T23:24:51.224Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e8 r14 = 0x00007fc866577048
[task 2019-07-21T23:24:51.226Z] 23:24:51 INFO - r15 = 0x00007fc86140a060 rip = 0x00007fc876187289
[task 2019-07-21T23:24:51.231Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.236Z] 23:24:51 INFO - 3 libxul.so!CorePS::Destroy(mozilla::BaseAutoLock<PSMutex&> const&) [platform.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 299 + 0x14]
[task 2019-07-21T23:24:51.237Z] 23:24:51 INFO - rbx = 0x00007fc866577040 rbp = 0x00007ffdf9d83550
[task 2019-07-21T23:24:51.238Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83540 r12 = 0x00007ffdf9d835a8
[task 2019-07-21T23:24:51.239Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e8 r14 = 0x00007ffdf9d83568
[task 2019-07-21T23:24:51.242Z] 23:24:51 INFO - r15 = 0x0000000000002f54 rip = 0x00007fc876173626
[task 2019-07-21T23:24:51.247Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.255Z] 23:24:51 INFO - 4 libxul.so!profiler_shutdown() [platform.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 3083 + 0x8]
[task 2019-07-21T23:24:51.256Z] 23:24:51 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffdf9d83590
[task 2019-07-21T23:24:51.257Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83560 r12 = 0x00007ffdf9d835a8
[task 2019-07-21T23:24:51.258Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e8 r14 = 0x00007ffdf9d83568
[task 2019-07-21T23:24:51.260Z] 23:24:51 INFO - r15 = 0x0000000000002f54 rip = 0x00007fc876173038
[task 2019-07-21T23:24:51.261Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.277Z] 23:24:51 INFO - 5 libxul.so!XRE_InitChildProcess(int, char**, XREChildData const*) [nsEmbedFunctions.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 769 + 0xa]
[task 2019-07-21T23:24:51.288Z] 23:24:51 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffdf9d83880
[task 2019-07-21T23:24:51.289Z] 23:24:51 INFO - rsp = 0x00007ffdf9d835a0 r12 = 0x00007ffdf9d835a8
[task 2019-07-21T23:24:51.290Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e8 r14 = 0x0000000000000000
[task 2019-07-21T23:24:51.290Z] 23:24:51 INFO - r15 = 0x0000000000002f54 rip = 0x00007fc87643f4cb
[task 2019-07-21T23:24:51.291Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.292Z] 23:24:51 INFO - 6 plugin-container!content_process_main(mozilla::Bootstrap*, int, char**) [plugin-container.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 56 + 0x12]
[task 2019-07-21T23:24:51.293Z] 23:24:51 INFO - rbx = 0x000000000000000f rbp = 0x00007ffdf9d838c0
[task 2019-07-21T23:24:51.294Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83890 r12 = 0x0000558cf5dbcab8
[task 2019-07-21T23:24:51.295Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e0 r14 = 0x00007ffdf9d839e8
[task 2019-07-21T23:24:51.298Z] 23:24:51 INFO - r15 = 0x00007fc866517650 rip = 0x0000558cf5dbcc48
[task 2019-07-21T23:24:51.307Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.309Z] 23:24:51 INFO - 7 plugin-container!main [MozillaRuntimeMain.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 23 + 0xb]
[task 2019-07-21T23:24:51.310Z] 23:24:51 INFO - rbx = 0x00007ffdf9d838d8 rbp = 0x00007ffdf9d83900
[task 2019-07-21T23:24:51.312Z] 23:24:51 INFO - rsp = 0x00007ffdf9d838d0 r12 = 0x0000558cf5dbcab8
[task 2019-07-21T23:24:51.314Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e0 r14 = 0x00007ffdf9d839e8
[task 2019-07-21T23:24:51.319Z] 23:24:51 INFO - r15 = 0x0000000000000010 rip = 0x0000558cf5dbccb7
[task 2019-07-21T23:24:51.323Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.324Z] 23:24:51 INFO - 8 libc-2.23.so + 0x20830
[task 2019-07-21T23:24:51.328Z] 23:24:51 INFO - rbx = 0x0000000000000000 rbp = 0x0000558cf5e2dae0
[task 2019-07-21T23:24:51.329Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83910 r12 = 0x0000558cf5dbcab8
[task 2019-07-21T23:24:51.330Z] 23:24:51 INFO - r13 = 0x00007ffdf9d839e0 r14 = 0x0000000000000000
[task 2019-07-21T23:24:51.332Z] 23:24:51 INFO - r15 = 0x0000000000000000 rip = 0x00007fc870d80830
[task 2019-07-21T23:24:51.333Z] 23:24:51 INFO - Found by: call frame info
[task 2019-07-21T23:24:51.334Z] 23:24:51 INFO - 9 plugin-container!content_process_main(mozilla::Bootstrap*, int, char**) [plugin-container.cpp:5e0d7f8b5620cbe84280e877b4d14c442925d1ba : 0 + 0x5]
[task 2019-07-21T23:24:51.339Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83930 rip = 0x0000558cf5dbcc71
[task 2019-07-21T23:24:51.355Z] 23:24:51 INFO - Found by: stack scanning
[task 2019-07-21T23:24:51.356Z] 23:24:51 INFO - 10 plugin-container + 0xbab8
[task 2019-07-21T23:24:51.365Z] 23:24:51 INFO - rsp = 0x00007ffdf9d83948 rip = 0x0000558cf5dbcab8
[task 2019-07-21T23:24:51.366Z] 23:24:51 INFO - Found by: stack scanning
[task 2019-07-21T23:24:51.375Z] 23:24:51 INFO - 11 ld-2.23.so + 0x107db
[task 2019-07-21T23:24:51.376Z] 23:24:51 INFO - rsp = 0x00007ffdf9d839a0 rip = 0x00007fc87b0bf7db
[task 2019-07-21T23:24:51.377Z] 23:24:51 INFO - Found by: stack scanning
[task 2019-07-21T23:24:51.380Z] 23:24:51 INFO - 12 plugin-container + 0xbab8
[task 2019-07-21T23:24:51.380Z] 23:24:51 INFO - rsp = 0x00007ffdf9d839b8 rip = 0x0000558cf5dbcab8
[task 2019-07-21T23:24:51.381Z] 23:24:51 INFO - Found by: stack scanning
[task 2019-07-21T23:24:51.381Z] 23:24:51 INFO - 13 plugin-container!_start + 0x29
[task 2019-07-21T23:24:51.382Z] 23:24:51 INFO - rsp = 0x00007ffdf9d839d0 rip = 0x0000558cf5dbcae1
[task 2019-07-21T23:24:51.400Z] 23:24:51 INFO - Found by: stack scanning

Gerald can you assign someone to take a look at this failure?

Flags: needinfo?(gsquelart)
Whiteboard: [stockwell needswork:owner]

I've searched commits in the couple of days before the first failure (1 June), but didn't see anything obvious.
Markus, you've worked on ProfilingStack::stackPointer, could you please have a look, or redirect to someone who may know?

Flags: needinfo?(gsquelart) → needinfo?(mstange)

This means that push/pop calls are unbalanced. Looking through the callers of ProfilingStack::pushLabelFrame/pushMarkerFrame/pushJSFrame, most callers use an RAII object to pair push and pop calls. There's one caller that does not, however: this call from the JavaScript interpreter. We have one GeckoProfilerThread::enter call in probes::EnterScript and two GeckoProfilerThread::exit calls in probes::ExitScript and in Interpret. The interpreter seems to be using some manual tracking to make sure that everything is in sync, but it is conceivable to me that this tracking might get confused.

In order to track down this bug, I recommend doing what Jeff suggested in comment 7: temporarily add some code that finds unbalanced push/pop calls in every one of the RAII helpers. This will cause us to crash much closer to the problem instead of at shutdown.

Flags: needinfo?(mstange)

Hi Gerald, can the recent occurrences here be from bug 1576819?

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=265564210&repo=autoland&lineNumber=13995

[task 2019-09-08T01:47:38.689Z] 01:47:38 INFO - TEST-START | dom/media/tests/mochitest/test_setSinkId.html
[task 2019-09-08T01:47:38.798Z] 01:47:38 INFO - GECKO(1377) | --DOCSHELL 0x7ff23eac8000 == 3 [pid = 1447] [id = {b08845c8-e832-4346-9a2b-a432494a49cc}] [url = about:blank]
[task 2019-09-08T01:47:38.819Z] 01:47:38 INFO - GECKO(1377) | [Parent 1377, Main Thread] WARNING: Cannot set transparency mode on non-popup windows.: file /builds/worker/workspace/build/src/widget/gtk/nsWindow.cpp, line 4443
[task 2019-09-08T01:47:38.859Z] 01:47:38 INFO - GECKO(1377) | ++DOMWINDOW == 10 (0x7ff242395c00) [pid = 1447] [serial = 149] [outer = 0x7ff24caf1f20]

[task 2019-09-08T01:47:41.257Z] 01:47:41 INFO - GECKO(1377) | [GPU 1403, Renderer] WARNING: Called GetMainThread but there isn't a main thread and we're not the main thread.: file /builds/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp, line 443
[task 2019-09-08T01:47:41.257Z] 01:47:41 INFO - GECKO(1377) | [GPU 1403, Renderer] WARNING: 'NS_FAILED(rv)', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp, line 241
[task 2019-09-08T01:47:41.257Z] 01:47:41 INFO - GECKO(1377) | [GPU 1403, Renderer] ###!!! ASSERTION: Failed NS_DispatchToMainThread() in shutdown; leaking: 'false', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp, line 243
[task 2019-09-08T01:48:02.321Z] 01:48:02 INFO - GECKO(1377) | #01: NS_DispatchToMainThread(nsIRunnable*, unsigned int) [xpcom/threads/nsThreadUtils.cpp:258]
[task 2019-09-08T01:48:02.321Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.321Z] 01:48:02 INFO - GECKO(1377) | Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/js/src/vm/ProfilingStack.cpp:21
[task 2019-09-08T01:48:02.321Z] 01:48:02 INFO - GECKO(1377) | #02: mozilla::wr::RenderThread::Release() [gfx/webrender_bindings/RenderThread.h:131]
[task 2019-09-08T01:48:02.321Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #03: mozilla::runnable_args_memfn<RefPtr<mozilla::wr::RenderThread>, void (mozilla::wr::RenderThread::)(mozilla::layers::SynchronousTask), mozilla::layers::SynchronousTask*>::~runnable_args_memfn() [media/mtransport/runnable_utils.h:140]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #04: mozilla::Runnable::Release() [xpcom/threads/nsThreadUtils.cpp:54]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #05: MessageLoop::RunTask(already_AddRefed<nsIRunnable>) [ipc/chromium/src/base/message_loop.cc:445]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #06: MessageLoop::DeferOrRunPendingTask(MessageLoop::PendingTask&&) [ipc/chromium/src/base/message_loop.cc:450]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #07: MessageLoop::DoWork() [ipc/chromium/src/base/message_loop.cc:523]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #08: base::MessagePumpDefault::Run(base::MessagePump::Delegate*) [ipc/chromium/src/base/message_pump_default.cc:35]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #09: MessageLoop::RunInternal() [ipc/chromium/src/base/message_loop.cc:315]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #10: MessageLoop::Run() [ipc/chromium/src/base/message_loop.cc:291]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #11: base::Thread::ThreadMain() [ipc/chromium/src/base/thread.cc:195]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #12: ThreadFunc(void*) [ipc/chromium/src/base/platform_thread_posix.cc:41]
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #13: libpthread.so.0 + 0x76ba
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.322Z] 01:48:02 INFO - GECKO(1377) | #14: libc.so.6 + 0x10741d
[task 2019-09-08T01:48:02.323Z] 01:48:02 INFO -
[task 2019-09-08T01:48:02.323Z] 01:48:02 INFO - GECKO(1377) | #15: ??? (???:???)

[task 2019-09-08T01:48:08.042Z] 01:48:08 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ ProfilingStack::~ProfilingStack()]
[task 2019-09-08T01:48:08.042Z] 01:48:08 INFO - Crash dump filename: /tmp/tmph20bI6.mozrunner/minidumps/76eedd25-0ec6-08bd-5df4-da4c1536441b.dmp
[task 2019-09-08T01:48:08.042Z] 01:48:08 INFO - Operating system: Linux
[task 2019-09-08T01:48:08.042Z] 01:48:08 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2019-09-08T01:48:08.043Z] 01:48:08 INFO - CPU: amd64
[task 2019-09-08T01:48:08.043Z] 01:48:08 INFO - family 6 model 85 stepping 4
[task 2019-09-08T01:48:08.043Z] 01:48:08 INFO - 1 CPU
[task 2019-09-08T01:48:08.043Z] 01:48:08 INFO -
[task 2019-09-08T01:48:08.044Z] 01:48:08 INFO - GPU: UNKNOWN
[task 2019-09-08T01:48:08.044Z] 01:48:08 INFO -
[task 2019-09-08T01:48:08.044Z] 01:48:08 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-09-08T01:48:08.044Z] 01:48:08 INFO - Crash address: 0x0
[task 2019-09-08T01:48:08.044Z] 01:48:08 INFO - Process uptime: not available
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO -
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - Thread 0 (crashed)
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - 0 libxul.so!ProfilingStack::~ProfilingStack() [ProfilingStack.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 21 + 0x29]
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - rax = 0x00007f9f188e954a rdx = 0x0000000000000000
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - rcx = 0x0000563ab0482c98 rbx = 0x00007f9f0bc4c270
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - rsi = 0x00007f9f2467e770 rdi = 0x00007f9f2467d540
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - rbp = 0x00007ffe3f3f6130 rsp = 0x00007ffe3f3f6130
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - r8 = 0x00007f9f2467e770 r9 = 0x00007f9f2574a740
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - r10 = 0x0000000000000000 r11 = 0x0000000000000000
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - r12 = 0x00007ffe3f3f6278 r13 = 0x00007ffe3f3f76b8
[task 2019-09-08T01:48:08.045Z] 01:48:08 INFO - r14 = 0x00007f9f24091908 r15 = 0x00007f9f0bc98e20
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rip = 0x00007f9f15587563
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - Found by: given as instruction pointer in context
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - 1 libxul.so!mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >::reset(RegisteredThread*) [UniquePtr.h:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 323 + 0x8]
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rbx = 0x00007f9f0bc4c270 rbp = 0x00007ffe3f3f6150
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f6140 r12 = 0x00007ffe3f3f6278
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b8 r14 = 0x00007f9f24091908
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - r15 = 0x00007f9f0bc98e20 rip = 0x00007f9f151a629b
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - 2 libxul.so!mozilla::Vector<mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >, 0ul, mozilla::MallocAllocPolicy>::~Vector() [Vector.h:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 902 + 0x10]
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rbx = 0x00007f9f0bc98e08 rbp = 0x00007ffe3f3f61a0
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f6160 r12 = 0x00007ffe3f3f6278
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b8 r14 = 0x00007f9f24091908
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - r15 = 0x00007f9f0bc98e20 rip = 0x00007f9f151a6a25
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - 3 libxul.so!CorePS::Destroy(mozilla::BaseAutoLock<PSMutex&> const&) [platform.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 296 + 0x14]
[task 2019-09-08T01:48:08.047Z] 01:48:08 INFO - rbx = 0x00007f9f24091900 rbp = 0x00007ffe3f3f61c0
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f61b0 r12 = 0x00007ffe3f3f6278
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b8 r14 = 0x00007ffe3f3f61e0
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - r15 = 0x0000000000000561 rip = 0x00007f9f15195e14
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - 4 libxul.so!profiler_shutdown() [platform.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 3069 + 0x8]
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffe3f3f6210
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f61d0 r12 = 0x00007ffe3f3f6278
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b8 r14 = 0x00007ffe3f3f61e0
[task 2019-09-08T01:48:08.048Z] 01:48:08 INFO - r15 = 0x0000000000000561 rip = 0x00007f9f151957d9
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - 5 libxul.so!XRE_InitChildProcess(int, char**, XREChildData const*) [nsEmbedFunctions.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 784 + 0xa]
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffe3f3f6510
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f6220 r12 = 0x00007ffe3f3f6278
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b8 r14 = 0x0000000000000000
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - r15 = 0x0000000000000561 rip = 0x00007f9f1546cbc3
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - 6 firefox-bin!content_process_main(mozilla::Bootstrap*, int, char**) [plugin-container.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 56 + 0x11]
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rbx = 0x0000000000000010 rbp = 0x00007ffe3f3f6550
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f6520 r12 = 0x00007ffe3f3f6520
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - r13 = 0x000000ebc3533986 r14 = 0x00007ffe3f3f76b8
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - r15 = 0x00007f9f24002730 rip = 0x0000563ab03e0d32
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - 7 firefox-bin!main [nsBrowserApp.cpp:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 272 + 0x19]
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rbx = 0x00007ffe3f3f65a0 rbp = 0x00007ffe3f3f75d0
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f6560 r12 = 0x00007ffe3f3f76b8
[task 2019-09-08T01:48:08.049Z] 01:48:08 INFO - r13 = 0x000000ebc3533986 r14 = 0x0000000000000011
[task 2019-09-08T01:48:08.050Z] 01:48:08 INFO - r15 = 0x0000563ab0482548 rip = 0x0000563ab03e1057
[task 2019-09-08T01:48:08.050Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.050Z] 01:48:08 INFO - 8 libc-2.23.so + 0x20830
[task 2019-09-08T01:48:08.050Z] 01:48:08 INFO - rbx = 0x0000000000000000 rbp = 0x0000563ab0457f70
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f75e0 r12 = 0x0000563ab03e0b98
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - r13 = 0x00007ffe3f3f76b0 r14 = 0x0000000000000000
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - r15 = 0x0000000000000000 rip = 0x00007f9f242d8830
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - Found by: call frame info
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - 9 firefox-bin!<name omitted> [UniquePtr.h:da3d3d8b83c8d9003ef6d29585af10f1db8a1151 : 274 + 0x17]
[task 2019-09-08T01:48:08.051Z] 01:48:08 INFO - rsp = 0x00007ffe3f3f7600 rip = 0x0000563ab03e0d75
[task 2019-09-08T01:48:08.052Z] 01:48:08 INFO - Found by: stack scanning

Flags: needinfo?(gsquelart)

(In reply to Andreea Pavel [:apavel] from comment #40)

Hi Gerald, can the recent occurrences here be from bug 1576819?

I don't think so: bug 1576819 only converted some function calls into macro equivalents, so there shouldn't be any actual compiled-code changes there.
Also that bug was touching ProfilerMarkerPayloads, while the issue here is with ProfilingStacks, which are not directly related.

I've already started working on a bug-finder patch (to try and catch the origin of the push/pop mismatch that ends up triggering this assertion), but I haven't had time yet to test it.
Since the occurrences have gone up, I'll try and work on it in the coming week...

(Clearing NI to notify Andreea; Adding NI:myself to keep it on my radar.)

Flags: needinfo?(gsquelart)

NI:myself again, see comment 41.

Flags: needinfo?(gsquelart)

Thank you!

There are 32 total failures in the last 7 days on:

*debug: windows10-64-qr, windows10-64, linux64-qr
*opt: windows10-64-shippable-qr, windows10-64

Recent failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=266802988&repo=autoland&lineNumber=10372

[task 2019-09-16T02:59:15.850Z] 02:59:15 INFO - TEST-START | Shutdown
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - Passed: 23649
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - Failed: 0
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - Todo: 82
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - Mode: e10s
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - Slowest: 91462ms - /tests/dom/html/test/test_fullscreen-api.html
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - SimpleTest FINISHED
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - TEST-INFO | Ran 1 Loops
[task 2019-09-16T02:59:15.851Z] 02:59:15 INFO - SimpleTest FINISHED

[task 2019-09-16T02:59:44.810Z] 02:59:44 INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ ProfilingStack::~ProfilingStack()]
[task 2019-09-16T02:59:44.810Z] 02:59:44 INFO - Crash dump filename: /tmp/tmpKjuCM1.mozrunner/minidumps/52a9115e-cef3-0699-bc10-8d0369c62e62.dmp
[task 2019-09-16T02:59:44.810Z] 02:59:44 INFO - Operating system: Linux
[task 2019-09-16T02:59:44.810Z] 02:59:44 INFO - 0.0.0 Linux 4.4.0-1014-aws #14taskcluster1-Ubuntu SMP Tue Apr 3 10:27:00 UTC 2018 x86_64
[task 2019-09-16T02:59:44.811Z] 02:59:44 INFO - CPU: amd64
[task 2019-09-16T02:59:44.811Z] 02:59:44 INFO - family 6 model 85 stepping 4
[task 2019-09-16T02:59:44.811Z] 02:59:44 INFO - 1 CPU
[task 2019-09-16T02:59:44.812Z] 02:59:44 INFO -
[task 2019-09-16T02:59:44.812Z] 02:59:44 INFO - GPU: UNKNOWN
[task 2019-09-16T02:59:44.813Z] 02:59:44 INFO -
[task 2019-09-16T02:59:44.813Z] 02:59:44 INFO - Crash reason: SIGSEGV /SEGV_MAPERR
[task 2019-09-16T02:59:44.814Z] 02:59:44 INFO - Crash address: 0x0
[task 2019-09-16T02:59:44.815Z] 02:59:44 INFO - Process uptime: not available
[task 2019-09-16T02:59:44.816Z] 02:59:44 INFO -
[task 2019-09-16T02:59:44.816Z] 02:59:44 INFO - Thread 0 (crashed)
[task 2019-09-16T02:59:44.817Z] 02:59:44 INFO - 0 libxul.so!ProfilingStack::~ProfilingStack() [ProfilingStack.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 21 + 0x29]
[task 2019-09-16T02:59:44.817Z] 02:59:44 INFO - rax = 0x00007f5ab369f56a rdx = 0x0000000000000000
[task 2019-09-16T02:59:44.817Z] 02:59:44 INFO - rcx = 0x000055bc96a48c98 rbx = 0x00007f5aa694b270
[task 2019-09-16T02:59:44.819Z] 02:59:44 INFO - rsi = 0x00007f5abf4a9770 rdi = 0x00007f5abf4a8540
[task 2019-09-16T02:59:44.819Z] 02:59:44 INFO - rbp = 0x00007ffc1ce37130 rsp = 0x00007ffc1ce37130
[task 2019-09-16T02:59:44.819Z] 02:59:44 INFO - r8 = 0x00007f5abf4a9770 r9 = 0x00007f5ac0575740
[task 2019-09-16T02:59:44.820Z] 02:59:44 INFO - r10 = 0x0000000000000000 r11 = 0x0000000000000000
[task 2019-09-16T02:59:44.820Z] 02:59:44 INFO - r12 = 0x00007ffc1ce37278 r13 = 0x00007ffc1ce386b8
[task 2019-09-16T02:59:44.821Z] 02:59:44 INFO - r14 = 0x00007f5abee91908 r15 = 0x00007f5a9d374120
[task 2019-09-16T02:59:44.821Z] 02:59:44 INFO - rip = 0x00007f5ab02c4df3
[task 2019-09-16T02:59:44.822Z] 02:59:44 INFO - Found by: given as instruction pointer in context
[task 2019-09-16T02:59:44.822Z] 02:59:44 INFO - 1 libxul.so!mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >::reset(RegisteredThread*) [UniquePtr.h:fb870f8c2eb458adb392c549f3bef422d483abf3 : 323 + 0x8]
[task 2019-09-16T02:59:44.822Z] 02:59:44 INFO - rbx = 0x00007f5aa694b270 rbp = 0x00007ffc1ce37150
[task 2019-09-16T02:59:44.823Z] 02:59:44 INFO - rsp = 0x00007ffc1ce37140 r12 = 0x00007ffc1ce37278
[task 2019-09-16T02:59:44.823Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b8 r14 = 0x00007f5abee91908
[task 2019-09-16T02:59:44.824Z] 02:59:44 INFO - r15 = 0x00007f5a9d374120 rip = 0x00007f5aafee3a63
[task 2019-09-16T02:59:44.825Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.825Z] 02:59:44 INFO - 2 libxul.so!mozilla::Vector<mozilla::UniquePtr<RegisteredThread, mozilla::DefaultDelete<RegisteredThread> >, 0ul, mozilla::MallocAllocPolicy>::~Vector() [Vector.h:fb870f8c2eb458adb392c549f3bef422d483abf3 : 902 + 0x10]
[task 2019-09-16T02:59:44.826Z] 02:59:44 INFO - rbx = 0x00007f5a9d374108 rbp = 0x00007ffc1ce371a0
[task 2019-09-16T02:59:44.827Z] 02:59:44 INFO - rsp = 0x00007ffc1ce37160 r12 = 0x00007ffc1ce37278
[task 2019-09-16T02:59:44.827Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b8 r14 = 0x00007f5abee91908
[task 2019-09-16T02:59:44.828Z] 02:59:44 INFO - r15 = 0x00007f5a9d374120 rip = 0x00007f5aafee41ed
[task 2019-09-16T02:59:44.829Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.829Z] 02:59:44 INFO - 3 libxul.so!CorePS::Destroy(mozilla::BaseAutoLock<PSMutex&> const&) [platform.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 296 + 0x14]
[task 2019-09-16T02:59:44.830Z] 02:59:44 INFO - rbx = 0x00007f5abee91900 rbp = 0x00007ffc1ce371c0
[task 2019-09-16T02:59:44.831Z] 02:59:44 INFO - rsp = 0x00007ffc1ce371b0 r12 = 0x00007ffc1ce37278
[task 2019-09-16T02:59:44.831Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b8 r14 = 0x00007ffc1ce371e0
[task 2019-09-16T02:59:44.832Z] 02:59:44 INFO - r15 = 0x0000000000000485 rip = 0x00007f5aafed34aa
[task 2019-09-16T02:59:44.833Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.834Z] 02:59:44 INFO - 4 libxul.so!profiler_shutdown() [platform.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 3069 + 0x8]
[task 2019-09-16T02:59:44.834Z] 02:59:44 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffc1ce37210
[task 2019-09-16T02:59:44.835Z] 02:59:44 INFO - rsp = 0x00007ffc1ce371d0 r12 = 0x00007ffc1ce37278
[task 2019-09-16T02:59:44.836Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b8 r14 = 0x00007ffc1ce371e0
[task 2019-09-16T02:59:44.836Z] 02:59:44 INFO - r15 = 0x0000000000000485 rip = 0x00007f5aafed2e6f
[task 2019-09-16T02:59:44.837Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.838Z] 02:59:44 INFO - 5 libxul.so!XRE_InitChildProcess(int, char**, XREChildData const*) [nsEmbedFunctions.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 784 + 0xa]
[task 2019-09-16T02:59:44.838Z] 02:59:44 INFO - rbx = 0x0000000000000000 rbp = 0x00007ffc1ce37510
[task 2019-09-16T02:59:44.839Z] 02:59:44 INFO - rsp = 0x00007ffc1ce37220 r12 = 0x00007ffc1ce37278
[task 2019-09-16T02:59:44.840Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b8 r14 = 0x0000000000000000
[task 2019-09-16T02:59:44.841Z] 02:59:44 INFO - r15 = 0x0000000000000485 rip = 0x00007f5ab01aa93b
[task 2019-09-16T02:59:44.841Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.842Z] 02:59:44 INFO - 6 firefox-bin!content_process_main(mozilla::Bootstrap*, int, char**) [plugin-container.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 56 + 0x11]
[task 2019-09-16T02:59:44.843Z] 02:59:44 INFO - rbx = 0x0000000000000010 rbp = 0x00007ffc1ce37550
[task 2019-09-16T02:59:44.843Z] 02:59:44 INFO - rsp = 0x00007ffc1ce37520 r12 = 0x00007ffc1ce37520
[task 2019-09-16T02:59:44.844Z] 02:59:44 INFO - r13 = 0x000003b6e90eba42 r14 = 0x00007ffc1ce386b8
[task 2019-09-16T02:59:44.845Z] 02:59:44 INFO - r15 = 0x00007f5abee02730 rip = 0x000055bc969a6d32
[task 2019-09-16T02:59:44.845Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.846Z] 02:59:44 INFO - 7 firefox-bin!main [nsBrowserApp.cpp:fb870f8c2eb458adb392c549f3bef422d483abf3 : 272 + 0x19]
[task 2019-09-16T02:59:44.847Z] 02:59:44 INFO - rbx = 0x00007ffc1ce375a0 rbp = 0x00007ffc1ce385d0
[task 2019-09-16T02:59:44.847Z] 02:59:44 INFO - rsp = 0x00007ffc1ce37560 r12 = 0x00007ffc1ce386b8
[task 2019-09-16T02:59:44.848Z] 02:59:44 INFO - r13 = 0x000003b6e90eba42 r14 = 0x0000000000000011
[task 2019-09-16T02:59:44.849Z] 02:59:44 INFO - r15 = 0x000055bc96a48548 rip = 0x000055bc969a7057
[task 2019-09-16T02:59:44.849Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.850Z] 02:59:44 INFO - 8 libc-2.23.so + 0x20830
[task 2019-09-16T02:59:44.850Z] 02:59:44 INFO - rbx = 0x0000000000000000 rbp = 0x000055bc96a1df70
[task 2019-09-16T02:59:44.851Z] 02:59:44 INFO - rsp = 0x00007ffc1ce385e0 r12 = 0x000055bc969a6b98
[task 2019-09-16T02:59:44.852Z] 02:59:44 INFO - r13 = 0x00007ffc1ce386b0 r14 = 0x0000000000000000
[task 2019-09-16T02:59:44.853Z] 02:59:44 INFO - r15 = 0x0000000000000000 rip = 0x00007f5abf103830
[task 2019-09-16T02:59:44.853Z] 02:59:44 INFO - Found by: call frame info
[task 2019-09-16T02:59:44.854Z] 02:59:44 INFO - 9 firefox-bin!<name omitted> [UniquePtr.h:fb870f8c2eb458adb392c549f3bef422d483abf3 : 274 + 0x17]
[task 2019-09-16T02:59:44.855Z] 02:59:44 INFO - rsp = 0x00007ffc1ce38600 rip = 0x000055bc969a6d75
[task 2019-09-16T02:59:44.856Z] 02:59:44 INFO - Found by: stack scanning

If PROFILING_STACK_FORCE_CHECK is #defined (on by default on Nightly, for now),
ProfilingStack pops are checked against what is expected.
In case of mismatch, the whole stack is fprint'ed from root to top, and also
recently-popped frames since the last push if present, before assert-crashing.

This is not a solution for bug 1445822, but a way to find the cause of the
intermittent crashes in the wild.

Almost all of these intermittents are for QR jobs. Looking at the logs, most threads have shut down at this point, but there are still Rayon threads waiting for work. Combined with the intermittent nature it makes me wonder about a race condition: WR worker thread pushes a label, then the main thread shuts down the profiler and we assert. Or something.

Also some of the logs have things like:

[task 2019-09-08T01:48:02.321Z] 01:48:02     INFO - GECKO(1377) | #01: NS_DispatchToMainThread(nsIRunnable*, unsigned int) [xpcom/threads/nsThreadUtils.cpp:258]
[task 2019-09-08T01:48:02.321Z] 01:48:02     INFO - 
[task 2019-09-08T01:48:02.321Z] 01:48:02     INFO - GECKO(1377) | Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/js/src/vm/ProfilingStack.cpp:21
[task 2019-09-08T01:48:02.321Z] 01:48:02     INFO - GECKO(1377) | #02: mozilla::wr::RenderThread::Release() [gfx/webrender_bindings/RenderThread.h:131]
[task 2019-09-08T01:48:02.321Z] 01:48:02     INFO - 
[task 2019-09-08T01:48:02.322Z] 01:48:02     INFO - GECKO(1377) | #03: mozilla::runnable_args_memfn<RefPtr<mozilla::wr::RenderThread>, void (mozilla::wr::RenderThread::*)(mozilla::layers::SynchronousTask*), mozilla::layers::SynchronousTask*>::~runnable_args_memfn() [media/mtransport/runnable_utils.h:140]

Jeff, any thoughts? Can we force these threads to shut down at an earlier point?

Flags: needinfo?(jmuizelaar)

Nical, is a better person to answer.

Flags: needinfo?(jmuizelaar) → needinfo?(nical.bugzilla)

Rayon doesn't provide a way to block until its threads are shut down. This used to cause the destruction of the wrokers to sometimes race with some code that runs during XPCOM's thread shutdown so we had to work around it by leaking the thread pool (the worker threads never get deleted in current Gecko).

We block until the work submitted to the thread pool is complete, and webrender is shut down synchronously with the rest of gfx so jobs on the rayon threads should never race with the profiler shutdown (in theory anyway).

The assertion [task 2019-09-08T01:47:41.257Z] 01:47:41 INFO - GECKO(1377) | [GPU 1403, Renderer] ###!!! ASSERTION: Failed NS_DispatchToMainThread() in shutdown; leaking: 'false', file /builds/worker/workspace/build/src/xpcom/threads/nsThreadUtils.cpp, line 243 looks more suspect. This suggests that the shutdown of the PGPU protocol is either too late or not blocking. We used to have a pretty involved shutdown dance for most ipdl protocols to ensure things happened synchronously and in the right order, but it looks like it has changed, I am not sure by what system these things are enforced now.

Flags: needinfo?(nical.bugzilla)

Gerald, considering the latest comments, how about landing a simpler patch first where the profiler code just dumps the following on shutdown if stackPointer > 0 before asserting in SpiderMonkey code (this all could be debug only because debug builds are also affected): (1) the thread id/name associated with the stack and stackPointer value and maybe (2) the things still on that stack (or just the top entry).

That way we can narrow it down to either a WebRender thread or main thread and go from there. (Given this mainly affects WebRender I'm not convinced it's an unbalanced stack.)

Attachment #9092983 - Attachment is obsolete: true

(In reply to Jan de Mooij [:jandem] from comment #51)

Gerald, considering the latest comments, how about landing a simpler patch first where the profiler code just dumps the following on shutdown if stackPointer > 0 before asserting in SpiderMonkey code (this all could be debug only because debug builds are also affected): (1) the thread id/name associated with the stack and stackPointer value and maybe (2) the things still on that stack (or just the top entry).

That way we can narrow it down to either a WebRender thread or main thread and go from there. (Given this mainly affects WebRender I'm not convinced it's an unbalanced stack.)

Jan:

I reduced my previous patch to only dump the ProfilingStack contents before the assertion in the destructor, and launched a Try:
https://treeherder.mozilla.org/#/jobs?repo=try&group_state=expanded&selectedJob=267925396&revision=8c7093e090e34c60448f598cd55069a1c02c8043

Success! Here's one dump before the crash:
https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=267925396&repo=try&lineNumber=30392

I forgot to output the thread id, sorry, but it shows that a thread's "MessagePumpDefault::Run" label is still alive. And if you look below at the crash report, most threads still there are rayon.

It's the end of the day here, please feel free to change the patch and Try again if you want to extract more information...

But assuming rayon threads are our problem, what to do next? Should we just disable the assertion anyway, as the robot in comment 53 suggests? It's not really my area there, so someone closer to that code should have a look anyway.

Flags: needinfo?(gsquelart) → needinfo?(jdemooij)

Thinking about it, the profiler should know that the thread has not unregistered itself yet, so maybe we could allow having dangling labels in this case? This way we'd still have the test to catch real issues when threads end properly before the profiler shuts down. WDYT?

(In reply to Gerald Squelart [:gerald] from comment #55)

Thinking about it, the profiler should know that the thread has not unregistered itself yet, so maybe we could allow having dangling labels in this case? This way we'd still have the test to catch real issues when threads end properly before the profiler shuts down. WDYT?

What bothers me about this bug: what if the thread that didn't shutdown yet will try to pop its label(s) at some point, that would be a UAF? Maybe not an issue in this case if we rely on the OS to terminate those thread(s) eventually.

Do you happen to know why the issue is intermittent? If it's the Rayon thread pool I'd expect this to be a perma-shutdown-assertion.

I'll look at this more tomorrow or on Wednesday, thanks for your work on this so far!

Whiteboard: [stockwell disable-recommended] → [stockwell needswork:owner]

OK I added some more logging. The thread that's still active when we shutdown the profiler has thread name "Renderer" so this one I think.

nical, can you take this from here? It's very easy to repro on Try.

Flags: needinfo?(jdemooij) → needinfo?(nical.bugzilla)

I have too many things in-flight right now so I won't look into this one in the short term. If anyone else wants to, the renderer is synchronously shutdown along with GPUParent, so that suggests that shutdown of the PGPU protocol is either too late or not blocking (see comment 50). Why that would be I don't know.

Flags: needinfo?(nical.bugzilla)

Jeff, is there someone else who can look at this? It's one of the top intermittents, appears to be WebRender specific, and reproduces easily on Try so it shouldn't require that much effort to track down more.

Flags: needinfo?(jmuizelaar)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork]

Hi Geoff, do you have any advice here?

Flags: needinfo?(gbrown)

There's nothing we can disable to avoid this failure; we'll need to wait for a fix.

Flags: needinfo?(gbrown)
Whiteboard: [stockwell disable-recommended] → [stockwell needswork][comment64]

I'm working on a possible fix now: Making the ProfilingStack a ref-counted object, so it will live as long as needed by the profiler and the registered thread, whichever survives the longest.
I'm not assigning the bug to myself just yet, in case someone else wants to fly in with a better solution.

When the profiler shuts down, it destroys all remaining RegisteredThread's,
including the embedded ProfilingStack that stores labels for each thread.
Now, it is possible that some threads may outlive the
profiler (e.g., Rayon threads, which cannot be synchronously shut down), so
there could be labels left when the ProfilingStack gets destroyed, which
triggers the stackPointer == 0 assertion in the destructor.

Removing the assertion would not be enough, because after that, the thread may
still try to pop labels (using a pointer to the now-destroyed ProfilingStack
stored in AutoProfilerLabel), causing a UAF.
AutoProfilerLabel could theoretically check that the profiler is still alive
before trying to pop labels, but that would add a significant cost every time
the ProfilingStack is accessed, because the profiler mutex would need to be
locked.

The solution here is to make ProfilingStack a separate heap object, owned
both by the profiler through the thread's RegisteredThread, and by the thread
itself through the thread-local sProfilingStackOwnerTLS.
The first owning link is severed when the thread unregisters itself, or when the
profiler shuts down, whichever happens first.
The second owning link is severed when the thread unregisters itself.
This way the ProfilingStack stays alive as long as needed by the thread, even
if it outlives the profiler.

Implementation detail: ProfilingStack is used in other places as a straight
object, so it cannot be made ref-counted itself. Instead we use
ProfilingStackOwner, a small ref-counted shell around it.

I've attached my attempt at ref-counting ProfilingStack, but I'm not sure it's solving the issue:
https://treeherder.mozilla.org/#/jobs?repo=try&selectedJob=271097389&revision=e87dcd34a5a43e9237a894be26cbfc243897aff1
shows one test with the same failing assertion.

I've started another try with Jan's and my debugging printfs, maybe it will tell us whether it's really the same issue (too-long-lived thread), or a "real" stack mismatch as originally thought:
https://treeherder.mozilla.org/#/jobs?repo=try&revision=a82abb9853fc6968338581801c443fa656b43906

The soft freeze starts soon... Should we try to push this through quickly (pending proper approval of course), or take more time and land something next week after the release?

It's the end of the day here, can someone please take over? Markus and/or Jan, could you please have a look if you're available?

Flags: needinfo?(mstange)
Flags: needinfo?(jdemooij)

Markus/Jan did you get a chance to take a look at the above?

(In reply to Gerald Squelart [:gerald] (he/him) from comment #68)

The soft freeze starts soon... Should we try to push this through quickly (pending proper approval of course), or take more time and land something next week after the release?

Sorry for the delay :/ I don't think it's urgent enough to get a fix in quickly before the merge (and we can always back port a fix).

mstange, any thoughts here, since you're familiar with both the graphics and profiler side?

Flags: needinfo?(jdemooij)

No thoughts, but it looks like Gerald is on a good track here. I don't see any debug output in the try pushes though. Do we think the ASAN failures are related? I don't see stacks for those either.

Flags: needinfo?(mstange)

I caught "GECKO(1905) | Assertion failure: stackPointer == 0, at /builds/worker/workspace/build/src/js/src/vm/ProfilingStack.cpp:21" in Pernosco. If it would be useful let me know and I can provide a link (please email, I don't receive bugmail anymore).

Gerald, can you email Kyle and take a look at this in Pernosco? Thanks!

Flags: needinfo?(jmuizelaar) → needinfo?(gsquelart)

(In reply to Markus Stange [:mstange] from comment #76)

Gerald, can you email Kyle and take a look at this in Pernosco? Thanks!

(As discussed)
I think we know the cause of this issue, so I already told Kyle not to spend resources on this, sorry!
Anyway, I will try and get my patch reviewed and landed shortly, and then if this still happens, hopefully Pernosco will help us with whatever issue may still linger...

Flags: needinfo?(gsquelart)

(In reply to Gerald Squelart [:gerald] (he/him) from comment #77)

(In reply to Markus Stange [:mstange] from comment #76)

Gerald, can you email Kyle and take a look at this in Pernosco? Thanks!

(As discussed)
I think we know the cause of this issue, so I already told Kyle not to spend resources on this, sorry!
Anyway, I will try and get my patch reviewed and landed shortly, and then if this still happens, hopefully Pernosco will help us with whatever issue may still linger...

There is no reviewer set for that patch, can you add one?

Flags: needinfo?(gsquelart)

I'm working on it, will repost when ready...

Flags: needinfo?(gsquelart)
Assignee: nobody → gsquelart
Attachment #9100850 - Attachment description: Bug 1445822 - Make ProfilingStack ref-counted, owned by both profiler and thread - → Bug 1445822 - Make ProfilingStack ref-counted, owned by both profiler and thread - r?mstange

Note that the stackPointer == 0 assertion would crash anyway, so this
shouldn't add crashes. But it should make these new crashes appear separate,
making them hopefully easier to find and analyze.

Depends on D49141

Hey Gerald, it looks like Bug 1596492 is a dupe of this bug. But there is a testcase with a STR(dom.paintWorklet.enabled set to true). I hope it can be helpful for you.

Pushed by gsquelart@mozilla.com:
https://hg.mozilla.org/integration/autoland/rev/f6a1c0a70553
Make ProfilingStack ref-counted, owned by both profiler and thread - r=mstange
https://hg.mozilla.org/integration/autoland/rev/d5c1f19ee6e5
Dump stack and crash when ProfilingStack is not empty on destruction - r=mstange

https://hg.mozilla.org/mozilla-central/rev/f6a1c0a70553
https://hg.mozilla.org/mozilla-central/rev/d5c1f19ee6e5

Status: REOPENED → RESOLVED
Closed: 5 years ago → 5 years ago
status-firefox72: --- → fixed
Resolution: --- → FIXED
Target Milestone: --- → mozilla72

These failures affect ESR68 too, but I don't think we want to go through the effort of trying to uplift these patches there given the minimal likelihood of anybody meaningfully using the profiler there anyway. Would you be opposed to just skipping these tests on Linux asan & debug there instead?

status-firefox70: --- → wontfix
status-firefox71: --- → wontfix
status-firefox-esr68: --- → affected
Flags: needinfo?(gsquelart)
Keywords: assertion

Agreed, I doubt the profiler would get much (any?) use on ESR, and in such a way as to trigger this issue; and affected processes would crash right near the end of shutdown, nothing of value should run at that point.

Do I/you/someone need to do something to skip these tests there?

Flags: needinfo?(gsquelart)

Turns out the test in question was actually skipped already via bug 1544244 and we haven't seen any occurrence on ESR68 since.

status-firefox-esr68: affected → fixed
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Creator:
Created:
Updated:
Size: