1557208 - AddressSanitizer: heap-use-after-free /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/LinkedList.h:288:12 in isInList

Status: RESOLVED FIXED

(Core :: Graphics: WebRender, defect, P2)

Description

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=250290790&repo=autoland&lineNumber=4130
Raw log: https://taskcluster-artifacts.net/Kh2Alh2HQRSGsFIv_9Pp3g/0/public/logs/live_backing.log
Th push: https://treeherder.mozilla.org/#/jobs?repo=autoland&resultStatus=success%2Cpending%2Crunning%2Ctestfailed%2Cbusted%2Cexception&revision=4a1e7d5a5b449e60ef39d8e67e2b5cb54d3d389d&searchStr=linux%2Cx64%2Cquantumrender%2Casan%2Copt%2Creftests%2Ctest-linux64-asan-qr%2Fopt-reftest-e10s-1%2Cr%28r1%29&selectedJob=250290790

[task 2019-06-06T02:13:21.137Z] 02:13:21    ERROR - ==5290==ERROR: AddressSanitizer: heap-use-after-free on address 0x61100002ffd8 at pc 0x7fe72dc4c798 bp 0x7fe724f14d40 sp 0x7fe724f14d38
[task 2019-06-06T02:13:21.138Z] 02:13:21     INFO - READ of size 8 at 0x61100002ffd8 thread T5 (WRWorker#0)
[task 2019-06-06T02:13:21.969Z] 02:13:21     INFO -     #0 0x7fe72dc4c797 in isInList /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/LinkedList.h:288:12
[task 2019-06-06T02:13:21.970Z] 02:13:21     INFO -     #1 0x7fe72dc4c797 in nsThread::MaybeRemoveFromThreadList() /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:401
[task 2019-06-06T02:13:21.970Z] 02:13:21     INFO -     #2 0x7fe72dc4e799 in nsThread::~nsThread() /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:633:3
[task 2019-06-06T02:13:21.970Z] 02:13:21     INFO -     #3 0x7fe72dc4ed6d in nsThread::~nsThread() /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:629:23
[task 2019-06-06T02:13:21.972Z] 02:13:21     INFO -     #4 0x7fe72dc4b78d in nsThread::Release() /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:187:1
[task 2019-06-06T02:13:21.979Z] 02:13:21     INFO -     #5 0x7fe74ec3a82d in _PR_DestroyThreadPrivate /builds/worker/workspace/build/src/nsprpub/pr/src/threads/prtpd.c:237:25
[task 2019-06-06T02:13:21.985Z] 02:13:21     INFO -     #6 0x7fe74ec2aa94 in _pt_thread_death_internal /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:855:9
[task 2019-06-06T02:13:21.986Z] 02:13:21     INFO -     #7 0x7fe74ec2ad16 in _pt_thread_death /builds/worker/workspace/build/src/nsprpub/pr/src/pthreads/ptthread.c:828:5
[task 2019-06-06T02:13:21.990Z] 02:13:21     INFO -     #8 0x7fe74e87d438 in __nptl_deallocate_tsd.part.4 (/lib/x86_64-linux-gnu/libpthread.so.0+0x6438)
[task 2019-06-06T02:13:21.991Z] 02:13:21     INFO -     #9 0x7fe74e87e86f in start_thread (/lib/x86_64-linux-gnu/libpthread.so.0+0x786f)
[task 2019-06-06T02:13:22.087Z] 02:13:22     INFO -     #10 0x7fe74d90741c in clone /build/glibc-LK5gWL/glibc-2.23/misc/../sysdeps/unix/sysv/linux/x86_64/clone.S:109
[task 2019-06-06T02:13:22.089Z] 02:13:22     INFO - 0x61100002ffd8 is located 24 bytes inside of 200-byte region [0x61100002ffc0,0x611000030088)
[task 2019-06-06T02:13:22.090Z] 02:13:22     INFO - freed by thread T0 (GPU Process) here:
[task 2019-06-06T02:13:22.092Z] 02:13:22     INFO -     #0 0x5557b3c6a182 in free /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:124:3
[task 2019-06-06T02:13:22.093Z] 02:13:22     INFO -     #1 0x7fe72dc4b78d in nsThread::Release() /builds/worker/workspace/build/src/xpcom/threads/nsThread.cpp:187:1
[task 2019-06-06T02:13:22.095Z] 02:13:22     INFO -     #2 0x7fe72dc5755c in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:46:40
[task 2019-06-06T02:13:22.096Z] 02:13:22     INFO -     #3 0x7fe72dc5755c in Release /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:363
[task 2019-06-06T02:13:22.098Z] 02:13:22     INFO -     #4 0x7fe72dc5755c in ~RefPtr /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/RefPtr.h:77
[task 2019-06-06T02:13:22.099Z] 02:13:22     INFO -     #5 0x7fe72dc5755c in Destruct /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:525
[task 2019-06-06T02:13:22.100Z] 02:13:22     INFO -     #6 0x7fe72dc5755c in DestructRange /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:2183
[task 2019-06-06T02:13:22.103Z] 02:13:22     INFO -     #7 0x7fe72dc5755c in ClearAndRetainStorage /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:1300
[task 2019-06-06T02:13:22.104Z] 02:13:22     INFO -     #8 0x7fe72dc5755c in ~nsTArray_Impl /builds/worker/workspace/build/src/obj-firefox/dist/include/nsTArray.h:881
[task 2019-06-06T02:13:22.105Z] 02:13:22     INFO -     #9 0x7fe72dc5755c in nsThreadManager::Shutdown() /builds/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp:318
[task 2019-06-06T02:13:22.106Z] 02:13:22     INFO -     #10 0x7fe72dca93a1 in mozilla::ShutdownXPCOM(nsIServiceManager*) /builds/worker/workspace/build/src/xpcom/build/XPCOMInit.cpp:649:28
[task 2019-06-06T02:13:22.107Z] 02:13:22     INFO -     #11 0x7fe739d51e45 in XRE_InitChildProcess(int, char**, XREChildData const*) /builds/worker/workspace/build/src/toolkit/xre/nsEmbedFunctions.cpp:753:16
[task 2019-06-06T02:13:22.108Z] 02:13:22     INFO -     #12 0x5557b3c9d3a7 in content_process_main /builds/worker/workspace/build/src/browser/app/../../ipc/contentproc/plugin-container.cpp:56:28
[task 2019-06-06T02:13:22.109Z] 02:13:22     INFO -     #13 0x5557b3c9d3a7 in main /builds/worker/workspace/build/src/browser/app/nsBrowserApp.cpp:263
[task 2019-06-06T02:13:22.111Z] 02:13:22     INFO -     #14 0x7fe74d82082f in __libc_start_main /build/glibc-LK5gWL/glibc-2.23/csu/../csu/libc-start.c:291
[task 2019-06-06T02:13:22.112Z] 02:13:22     INFO - previously allocated by thread T5 (WRWorker#0) here:
[task 2019-06-06T02:13:22.116Z] 02:13:22     INFO -     #0 0x5557b3c6a503 in __interceptor_malloc /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_malloc_linux.cc:146:3
[task 2019-06-06T02:13:22.117Z] 02:13:22     INFO -     #1 0x5557b3c9ef9d in moz_xmalloc /builds/worker/workspace/build/src/memory/mozalloc/mozalloc.cpp:52:15
[task 2019-06-06T02:13:22.121Z] 02:13:22     INFO -     #2 0x7fe72dc50817 in operator new /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/mozalloc.h:144:10
[task 2019-06-06T02:13:22.122Z] 02:13:22     INFO -     #3 0x7fe72dc50817 in nsThreadManager::GetCurrentThread() /builds/worker/workspace/build/src/xpcom/threads/nsThreadManager.cpp:376
[task 2019-06-06T02:13:22.158Z] 02:13:22     INFO -     #4 0x7fe739492615 in profiler_register_thread(char const*, void*) /builds/worker/workspace/build/src/tools/profiler/core/platform.cpp:3666:9
[task 2019-06-06T02:13:22.199Z] 02:13:22     INFO -     #5 0x7fe730557f1f in gecko_profiler_register_thread /builds/worker/workspace/build/src/gfx/layers/wr/WebRenderBridgeParent.cpp:142:3
[task 2019-06-06T02:13:22.221Z] 02:13:22     INFO -     #6 0x7fe73c078772 in webrender_bindings::bindings::wr_thread_pool_new::_$u7b$$u7b$closure$u7d$$u7d$::h9e0a5abfefaf5976 /builds/worker/workspace/build/src/gfx/webrender_bindings/src/bindings.rs:1048:12
[task 2019-06-06T02:13:22.222Z] 02:13:22     INFO - Thread T5 (WRWorker#0) created by T0 (GPU Process) here:
[task 2019-06-06T02:13:22.239Z] 02:13:22     INFO -     #0 0x5557b3c52add in pthread_create /builds/worker/workspace/moz-toolchain/src/llvm/projects/compiler-rt/lib/asan/asan_interceptors.cc:210:3
[task 2019-06-06T02:13:22.249Z] 02:13:22     INFO -     #1 0x7fe73c6cebe5 in std::sys::unix::thread::Thread::new::hba7601f1ccb9f089 /rustc/3c235d5600393dfe6c36eeed34042efad8d4f26e/src/libstd/sys/unix/thread.rs:68:18
[task 2019-06-06T02:13:22.251Z] 02:13:22     INFO - SUMMARY: AddressSanitizer: heap-use-after-free /builds/worker/workspace/build/src/obj-firefox/dist/include/mozilla/LinkedList.h:288:12 in isInList
[task 2019-06-06T02:13:22.251Z] 02:13:22     INFO - Shadow bytes around the buggy address:
[task 2019-06-06T02:13:22.253Z] 02:13:22     INFO -   0x0c227fffdfa0: fa fa fa fa fa fa fa fa fd fd fd fd fd fd fd fd
[task 2019-06-06T02:13:22.254Z] 02:13:22     INFO -   0x0c227fffdfb0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
[task 2019-06-06T02:13:22.256Z] 02:13:22     INFO -   0x0c227fffdfc0: fd fd fd fd fa fa fa fa fa fa fa fa fa fa fa fa
[task 2019-06-06T02:13:22.258Z] 02:13:22     INFO -   0x0c227fffdfd0: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
[task 2019-06-06T02:13:22.260Z] 02:13:22     INFO -   0x0c227fffdfe0: fd fd fd fd fd fd fd fd fd fd fd fd fa fa fa fa
[task 2019-06-06T02:13:22.261Z] 02:13:22     INFO - =>0x0c227fffdff0: fa fa fa fa fa fa fa fa fd fd fd[fd]fd fd fd fd
[task 2019-06-06T02:13:22.263Z] 02:13:22     INFO -   0x0c227fffe000: fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd fd
[task 2019-06-06T02:13:22.264Z] 02:13:22     INFO -   0x0c227fffe010: fd fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[task 2019-06-06T02:13:22.266Z] 02:13:22     INFO -   0x0c227fffe020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
[task 2019-06-06T02:13:22.267Z] 02:13:22     INFO -   0x0c227fffe030: 00 00 00 00 00 00 00 00 00 00 00 fa fa fa fa fa
[task 2019-06-06T02:13:22.269Z] 02:13:22     INFO -   0x0c227fffe040: fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa fa
[task 2019-06-06T02:13:22.270Z] 02:13:22     INFO - Shadow byte legend (one shadow byte represents 8 application bytes):
[task 2019-06-06T02:13:22.272Z] 02:13:22     INFO -   Addressable:           00
[task 2019-06-06T02:13:22.274Z] 02:13:22     INFO -   Partially addressable: 01 02 03 04 05 06 07
[task 2019-06-06T02:13:22.275Z] 02:13:22     INFO -   Heap left redzone:       fa
[task 2019-06-06T02:13:22.277Z] 02:13:22     INFO -   Freed heap region:       fd
[task 2019-06-06T02:13:22.279Z] 02:13:22     INFO -   Stack left redzone:      f1
[task 2019-06-06T02:13:22.280Z] 02:13:22     INFO -   Stack mid redzone:       f2
[task 2019-06-06T02:13:22.282Z] 02:13:22     INFO -   Stack right redzone:     f3
[task 2019-06-06T02:13:22.290Z] 02:13:22     INFO -   Stack after return:      f5
[task 2019-06-06T02:13:22.291Z] 02:13:22     INFO -   Stack use after scope:   f8
[task 2019-06-06T02:13:22.292Z] 02:13:22     INFO -   Global redzone:          f9
[task 2019-06-06T02:13:22.293Z] 02:13:22     INFO -   Global init order:       f6
[task 2019-06-06T02:13:22.294Z] 02:13:22     INFO -   Poisoned by user:        f7
[task 2019-06-06T02:13:22.295Z] 02:13:22     INFO -   Container overflow:      fc
[task 2019-06-06T02:13:22.296Z] 02:13:22     INFO -   Array cookie:            ac
[task 2019-06-06T02:13:22.297Z] 02:13:22     INFO -   Intra object redzone:    bb
[task 2019-06-06T02:13:22.301Z] 02:13:22     INFO -   ASan internal:           fe
[task 2019-06-06T02:13:22.302Z] 02:13:22     INFO -   Left alloca redzone:     ca
[task 2019-06-06T02:13:22.303Z] 02:13:22     INFO -   Right alloca redzone:    cb
[task 2019-06-06T02:13:22.304Z] 02:13:22     INFO -   Shadow gap:              cc
[task 2019-06-06T02:13:22.305Z] 02:13:22     INFO - ==5290==ABORTING
[task 2019-06-06T02:13:22.447Z] 02:13:22     INFO - REFTEST INFO | Process mode: e10s
[task 2019-06-06T02:13:22.448Z] 02:13:22  WARNING - leakcheck | refcount logging is off, so leaks can't be detected!
[task 2019-06-06T02:13:22.448Z] 02:13:22     INFO - REFTEST INFO | Running tests in file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/svg/moz-only/reftest.list
[task 2019-06-06T02:13:22.468Z] 02:13:22     INFO - REFTEST INFO | INFO | runtests.py | ASan using symbolizer at /builds/worker/workspace/build/application/firefox/llvm-symbolizer
[task 2019-06-06T02:13:22.484Z] 02:13:22     INFO - REFTEST INFO | INFO | runtests.py | ASan running in default memory configuration
[task 2019-06-06T02:13:22.484Z] 02:13:22     INFO - REFTEST INFO | Running with e10s: True
[task 2019-06-06T02:13:22.486Z] 02:13:22     INFO - REFTEST INFO | Application command: /builds/worker/workspace/build/application/firefox/firefox -marionette -profile /tmp/tmpzZD0p9.mozrunner
[task 2019-06-06T02:13:24.017Z] 02:13:24     INFO - 1559787204009	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: mozillaAddons
[task 2019-06-06T02:13:24.023Z] 02:13:24     INFO - 1559787204010	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: telemetry
[task 2019-06-06T02:13:24.031Z] 02:13:24     INFO - 1559787204012	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: resource://pdf.js/
[task 2019-06-06T02:13:24.038Z] 02:13:24     INFO - 1559787204012	[e-mail]	WARN	Loading extension '[e-mail]': Reading manifest: Invalid extension permission: about:reader*
[task 2019-06-06T02:13:24.499Z] 02:13:24     INFO - 1559787204493	Marionette	TRACE	Received observer notification profile-after-change
[task 2019-06-06T02:13:24.758Z] 02:13:24     INFO - 1559787204748	Marionette	TRACE	Received observer notification command-line-startup
[task 2019-06-06T02:13:24.765Z] 02:13:24     INFO - 1559787204749	Marionette	TRACE	Received observer notification nsPref:changed
[task 2019-06-06T02:13:24.772Z] 02:13:24     INFO - 1559787204750	Marionette	DEBUG	Init aborted (running=false, enabled=true, finalUIStartup=false)
[task 2019-06-06T02:13:25.258Z] 02:13:25     INFO - 1559787205256	Marionette	TRACE	Received observer notification toplevel-window-ready
[task 2019-06-06T02:13:37.253Z] 02:13:37     INFO - 1559787217247	Marionette	TRACE	Received observer notification marionette-startup-requested
[task 2019-06-06T02:13:37.253Z] 02:13:37     INFO - 1559787217249	Marionette	TRACE	Waiting until startup recorder finished recording startup scripts...
[task 2019-06-06T02:13:37.534Z] 02:13:37     INFO - 1559787217530	Marionette	TRACE	All scripts recorded.
[task 2019-06-06T02:13:37.737Z] 02:13:37     INFO - 1559787217732	Marionette	INFO	Listening on port 2828
[task 2019-06-06T02:13:37.738Z] 02:13:37     INFO - 1559787217733	Marionette	DEBUG	Remote service is active
[task 2019-06-06T02:13:37.772Z] 02:13:37     INFO - 1559787217768	Marionette	DEBUG	Accepted connection 0 from 127.0.0.1:57708
[task 2019-06-06T02:13:37.829Z] 02:13:37     INFO - 1559787217820	Marionette	DEBUG	Accepted connection 1 from 127.0.0.1:57710
[task 2019-06-06T02:13:37.830Z] 02:13:37     INFO - 1559787217823	Marionette	DEBUG	Closed connection 0
[task 2019-06-06T02:13:37.853Z] 02:13:37     INFO - 1559787217851	Marionette	DEBUG	1 -> [0,1,"WebDriver:NewSession",{"strictFileInteractability":true}]
[task 2019-06-06T02:13:37.990Z] 02:13:37     INFO - 1559787217980	Marionette	TRACE	[4294967297] Frame script loaded
[task 2019-06-06T02:13:38.011Z] 02:13:38     INFO - 1559787218002	Marionette	TRACE	[4294967297] Frame script registered
[task 2019-06-06T02:13:38.040Z] 02:13:38     INFO - 1559787218036	Marionette	TRACE	[4294967300] Frame script loaded
[task 2019-06-06T02:13:38.040Z] 02:13:38     INFO - 1559787218037	Marionette	DEBUG	1 <- [1,1,null,{"sessionId":"8ef7ae00-bac2-49fc-8a79-775cc7e50db3","capabilities":{"browserName":"firefox","browserVersion":"69.0a ... p/tmpzZD0p9.mozrunner","moz:shutdownTimeout":180000,"moz:useNonSpecCompliantPointerOrigin":false,"moz:webdriverClick":true}}]
[task 2019-06-06T02:13:38.061Z] 02:13:38     INFO - 1559787218058	Marionette	DEBUG	1 -> [0,2,"Addon:Install",{"path":"/builds/worker/workspace/build/tests/reftest/reftest","temporary":true}]
[task 2019-06-06T02:13:38.081Z] 02:13:38     INFO - 1559787218073	Marionette	TRACE	[4294967300] Frame script registered
[task 2019-06-06T02:13:38.452Z] 02:13:38     INFO - 1559787218449	Marionette	DEBUG	1 <- [1,2,null,{"value":"[e-mail]"}]
[task 2019-06-06T02:13:38.811Z] 02:13:38     INFO - 1559787218803	Marionette	DEBUG	1 -> [0,3,"WebDriver:DeleteSession",{}]
[task 2019-06-06T02:13:38.813Z] 02:13:38     INFO - 1559787218810	Marionette	DEBUG	1 <- [1,3,null,{"value":null}]
[task 2019-06-06T02:13:38.914Z] 02:13:38     INFO - 1559787218910	Marionette	DEBUG	Closed connection 1
[task 2019-06-06T02:13:39.349Z] 02:13:39     INFO - ###!!! [Parent][RunMessage] Error: Channel closing: too late to send/recv, messages will be lost
[task 2019-06-06T02:13:39.808Z] 02:13:39     INFO - REFTEST TEST-START | file:///builds/worker/workspace/build/tests/reftest/tests/layout/reftests/svg/moz-only/xbl-basic-01.svg == file:///builds/worker/workspace/build/tests/reftest

Comment 1

[Jessie [:jbonisteel] pls NI]

Any suggestions about this bug?

Comment 3

[Andrew McCreight [:mccr8]]

Bug 1479743 looks like a debug version of this crash, or a similar crash with different timing. I'm not sure if this is a bug with webrender code or XPCOM code. The former seems more likely if these are all crashes in the webrender process with webrender threads, but who knows.

Comment 4

[Nicolas Silva [:nical]]

When we register WebRender's thread with the profiler we call nsThreadManager::GetCurrentThread which notices that the thread isn't registered with the xpcom's thread manager and lazily registers it. Unfortunately that's not safe to do because webrender/rayon own their threads and don't know about nsThredManager retaining references to it.

I suspect other factors make things a little more subtle than that because It's kind of crazy that we're only noticing this now.

Edit: Actually the registering/unregistering gymnastics appear to properly add/remove the nsThread wrapper. The problem is that the thread is destroyed after xpcom's shutdown which kills all remaing threads.

Comment 5

[Nicolas Silva [:nical]]

So I see two ways to go about this:

A) we can somehow avoid the requirement for an nsThread wrapper around threads that the profiler knows about.
B) we really need the nsThread wrapper in which case there is a dependency between all webrender thread lifetimes an ShutDownXPCOM, in which case we need to make sure webrender is destroyed synchronously (not cheap, not easy but I have a feeling this is how that's going to end).

Comment 7

[Eric Rahm [:erahm]]

(In reply to Nicolas Silva [:nical] from comment #5)

So I see two ways to go about this:

A) we can somehow avoid the requirement for an nsThread wrapper around threads that the profiler knows about.
B) we really need the nsThread wrapper in which case there is a dependency between all webrender thread lifetimes an ShutDownXPCOM, in which case we need to make sure webrender is destroyed synchronously (not cheap, not easy but I have a feeling this is how that's going to end).

B) is probably the way to go. We're seeing a lot of duplicates and see alsos of this, do you have time to look into a proper fix?

Comment 9

[Nicolas Silva [:nical]]

B) is not in the proper fix category. I'll make time, though if we really are in a hurry (I don't think we should) there is the option to not unregister the profiler in webrender threads until we figure out a way to synchronously track and shut down every resource that maintains webrender contexts alive (it'd likely leak a few things).

Comment 10

[Andrew McCreight [:mccr8]]

Any chance you can take a look at this, Nicolas? If not, do you know who else might be able to? There's more and more variants of this showing up on TreeHerder. Bug 1489014 and bug 1479273 are the latest.

Comment 11

[Jeff Muizelaar [:jrmuizel]]

So do we have any idea why this started showing up? As I understand it the issue creating nsThreads for webrender workers happened in bug 1476405. Shouldn't the problem have begun then?

Comment 12

[Jeff Muizelaar [:jrmuizel]]

Instead of having an nsThread wrapper for all of the threads and inflicting shutdown pain couldn't we just have a way to register/unregister thread stack sizes for threads that don't have nsThreads?

Comment 13

[Nicolas Silva [:nical]]

I'm back from PTO and looking into this again. Jeff's suggestion of manually registering/unregistering thread stack sizes (with something like a global atomic) for non-nsThreads would be a ton simpler and more robust than trying to enforce synchronousness and hunt all live resources keeping webrender contexts alive during shutdown.

Comment 14

[Nicolas Silva [:nical]]

Attachment: Bug 1557208 - Eagerly clear remaining documents during shutdown. r=sotaro

Comment 15

[Nicolas Silva [:nical]]

Attachment: Bug 1557208 - Block the caller during RenderBackend shutdown. r=sotaro

Comment 16

[Nicolas Silva [:nical]]

Attachment: Bug 1557208 - Deallocate thread pool handles during RenderThread sync shutdown. r=sotaro

Comment 17

[Nicolas Silva [:nical]]

Comment on attachment 9075324 [details]
Bug 1557208 - Eagerly clear remaining documents during shutdown. r=sotaro

Security Approval Request

• How easily could an exploit be constructed based on the patch?: Hard.
• Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
• Which older supported branches are affected by this flaw?: 67
• If not all supported branches, which bug introduced the flaw?: None
• Do you have backports for the affected branches?: No
• If not, how different, hard to create, and risky will they be?: Should not be hard, but probably not necessary during soft freeze. The main motivation is reducing intermittent failures on try, I don't think this is realistically exploitable. I would at least let the patches bake for a few days in nightly before uplifting.
• How likely is this patch to cause regressions; how much testing does it need?: Not very likely, let's let it bake in nightly for a week.

Comment 19

[Al Billings [:abillings - ex-MoCo]]

sec-approval+ for trunk

Comment 20

[Andreea Pavel [:apavel]]

Backed out for WR build bustages

Push with failures: https://treeherder.mozilla.org/#/jobs?repo=mozilla-inbound&selectedJob=255699679&resultStatus=testfailed%2Cbusted%2Cexception&revision=e5b264a27fa5bb7589c4264626b1c9082f3a41e5

Failure log: https://treeherder.mozilla.org/logviewer.html#/jobs?job_id=255699679&repo=mozilla-inbound&lineNumber=2598

Backout: https://hg.mozilla.org/integration/mozilla-inbound/rev/90e0236885939fd0f2e5de65807b613576e826cf

Comment 21

[Nicolas Silva [:nical]]

https://hg.mozilla.org/integration/mozilla-inbound/rev/3c73281b494c2dd19ea9eb3fa45ade53c7c75ba2
https://hg.mozilla.org/integration/mozilla-inbound/rev/eeb7e03c2541e454d2144c1f9dd5a58c2980aac3
https://hg.mozilla.org/integration/mozilla-inbound/rev/57284968eab1434aeb7a435082f0a7b59add13af

Comment 22

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/57284968eab1
https://hg.mozilla.org/mozilla-central/rev/eeb7e03c2541
https://hg.mozilla.org/mozilla-central/rev/3c73281b494c

Comment 24

[Nicolas Silva [:nical]]

Attachment: Bug 1557208 - Leak WebRender's thread pool. r=jrmuizel

Comment 25

[Nicolas Silva [:nical]]

Comment on attachment 9078771 [details]
Bug 1557208 - Leak WebRender's thread pool. r=jrmuizel

Security Approval Request

• How easily could an exploit be constructed based on the patch?: Hard.
• Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
• Which older supported branches are affected by this flaw?:
• If not all supported branches, which bug introduced the flaw?: None
• Do you have backports for the affected branches?: No
• If not, how different, hard to create, and risky will they be?: The patch should apply cleanly. It's not a risky patch.
• How likely is this patch to cause regressions; how much testing does it need?: Unlikely. Let's just let it bake in nightly for a bit. I'm not worried about the security aspect as much as the annoyance of intermittent failures on the CI.

Comment 26

[Al Billings [:abillings - ex-MoCo]]

Sec-approval+ for trunk.
I noted that ESR60 is unaffected but 68 is won't fix, which means ESR68 is affected.

We should get this on Beta and ESR68 once this bakes.

Comment 27

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/integration/mozilla-inbound/rev/927c1651317bd5e7c69635a48924643e81bc3079

Comment 28

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/mozilla-central/rev/927c1651317b

Comment 29

[Ryan VanderMeulen [:RyanVM]]

Please nominate this for Beta approval when you get a chance. IIUC, we don't really need this on ESR68 since WR is disabled by default there anyway?

Comment 30

[Nicolas Silva [:nical]]

Comment on attachment 9078771 [details]
Bug 1557208 - Leak WebRender's thread pool. r=jrmuizel

Beta/Release Uplift Approval Request

• User impact if declined: intermittent shutdown crashes. Probably not as impactful for users as it is for CI oranges.
• Is this code covered by automated tests?: No
• Has the fix been verified in Nightly?: Yes
• Needs manual test from QE?: No
• If yes, steps to reproduce:
• List of other uplifts needed: None
• Risk to taking this patch: Low
• Why is the change risky/not risky? (and alternatives if risky): Just leaking a unique resource that is normally alive until shutdown.
• String changes made/needed: None.

Comment 31

[Ryan VanderMeulen [:RyanVM]]

Comment on attachment 9078771 [details]
Bug 1557208 - Leak WebRender's thread pool. r=jrmuizel

Fixes a WebRender orange. Approved for 69.0b8. Note that per nical, only this last patch needs uplift and not the first 3.

Comment 32

[Sebastian Hengst [:aryx] (needinfo me if it's about an intermittent or backout)]

https://hg.mozilla.org/releases/mozilla-beta/rev/4aab05ffedb6