Closed Bug 1744081 Opened 3 years ago Closed 3 years ago

Intermittent Main app process exited normally | application crashed [@ je_free(void*)] from MOZ_RELEASE_ASSERT((run->mRegionsMask[elm] & (1U << bit)) == 0) (Double-free?)

Categories

(Core :: WebRTC: Audio/Video, defect, P1)

defect

Tracking

()

RESOLVED FIXED
97 Branch
Tracking Status
firefox-esr91 --- unaffected
firefox95 --- unaffected
firefox96 --- fixed
firefox97 --- fixed

People

(Reporter: intermittent-bug-filer, Assigned: pehrsons)

References

(Regression)

Details

(5 keywords, Whiteboard: [sec-survey])

Crash Data

Attachments

(1 file)

Filed by: archaeopteryx [at] coole-files.de
Parsed log: https://treeherder.mozilla.org/logviewer?job_id=359966726&repo=try
Full log: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/VWKzuLfUSOamva668WZh-Q/runs/0/artifacts/public/logs/live_backing.log


[task 2021-12-02T13:40:42.615Z] 13:40:42     INFO - GECKO(5292) | (ice/ERR) ICE(PC:{cea28181-d3dd-4b4f-aec0-ed961937191b} 1638452442445000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc): peer (PC:{cea28181-d3dd-4b4f-aec0-ed961937191b} 1638452442445000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc:default), stream(PC:{cea28181-d3dd-4b4f-aec0-ed961937191b} 1638452442445000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc transport-id=transport_0 - e8ccdf6f:acfa3de3bc7f6cf2aaacf2334627b762) tried to trickle ICE in inappropriate state 4
[task 2021-12-02T13:40:42.615Z] 13:40:42     INFO - GECKO(5292) | [Child 6188: Socket Thread]: D/mtransport Trickle candidate is redundant for stream 'PC:{cea28181-d3dd-4b4f-aec0-ed961937191b} 1638452442445000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc transport-id=transport_0' because it is completed:
[task 2021-12-02T13:40:42.616Z] 13:40:42     INFO - GECKO(5292) | [Child 6188: Socket Thread]: I/mtransport Flow[transport_0(none)]; Layer[dtls]: ****** SSL handshake completed ******
[task 2021-12-02T13:40:42.617Z] 13:40:42     INFO - GECKO(5292) | [Child 6188: Socket Thread]: I/mtransport Flow[transport_0(none)]; Layer[dtls]: Selected ALPN string: webrtc
[task 2021-12-02T13:40:42.617Z] 13:40:42     INFO - GECKO(5292) | [Child 6188: Socket Thread]: D/mtransport Created SRTP flow!
[task 2021-12-02T13:40:42.679Z] 13:40:42     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:43.213Z] 13:40:43     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:43.238Z] 13:40:43     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:43.257Z] 13:40:43     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:43.268Z] 13:40:43     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:43.677Z] 13:40:43     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:44.680Z] 13:40:44     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:45.266Z] 13:40:45     INFO - GECKO(5292) | [Child 6188: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2112: CloseInt: Closing PeerConnectionImpl {cea28181-d3dd-4b4f-aec0-ed961937191b}; ending call
[task 2021-12-02T13:40:45.273Z] 13:40:45     INFO - GECKO(5292) | [Child 6188: Main Thread]: I/jsep [{cea28181-d3dd-4b4f-aec0-ed961937191b} 1638452442445000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc]: stable -> closed
[task 2021-12-02T13:40:45.274Z] 13:40:45     INFO - GECKO(5292) | [Child 6188: Main Thread]: E/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2915: GetStats: Found no pipelines matching selector.
[task 2021-12-02T13:40:45.275Z] 13:40:45     INFO - GECKO(5292) | [Child 6188: Main Thread]: I/signaling [main|PeerConnectionImpl] PeerConnectionImpl.cpp:2112: CloseInt: Closing PeerConnectionImpl {83f6b66d-1368-40c1-b78a-7b417a685931}; ending call
[task 2021-12-02T13:40:45.275Z] 13:40:45     INFO - GECKO(5292) | [Child 6188: Main Thread]: I/jsep [{83f6b66d-1368-40c1-b78a-7b417a685931} 1638452442446000 (id=4294967686 url=https://example.com/tests/dom/media/webrtc/tests/moc]: stable -> closed
[task 2021-12-02T13:40:45.351Z] 13:40:45     INFO - GECKO(5292) | [Parent 6584, IPC I/O Parent] WARNING: pipe error: 232: file /builds/worker/checkouts/gecko/ipc/chromium/src/chrome/common/ipc_channel_win.cc:565
[task 2021-12-02T13:40:45.369Z] 13:40:45    ERROR - GECKO(5292) | A content process crashed and MOZ_CRASHREPORTER_SHUTDOWN is set, shutting down
[task 2021-12-02T13:40:45.420Z] 13:40:45     INFO - GECKO(5292) | 1638452445422	Marionette	TRACE	Received observer notification quit-application
[task 2021-12-02T13:40:45.424Z] 13:40:45     INFO - GECKO(5292) | 1638452445422	Marionette	INFO	Stopped listening on port 2828
[task 2021-12-02T13:40:45.425Z] 13:40:45     INFO - GECKO(5292) | 1638452445423	Marionette	DEBUG	Marionette stopped listening
[task 2021-12-02T13:40:45.438Z] 13:40:45     INFO - GECKO(5292) | [Parent 6584, IPC I/O Parent] WARNING: file /builds/worker/checkouts/gecko/ipc/chromium/src/base/process_util_win.cc:167
[task 2021-12-02T13:40:45.767Z] 13:40:45     INFO - TEST-INFO | Main app process: exit 0
[task 2021-12-02T13:40:45.769Z] 13:40:45     INFO - runtests.py | Application ran for: 0:05:42.377182
[task 2021-12-02T13:40:45.769Z] 13:40:45     INFO - zombiecheck | Reading PID log: C:\Users\task_163844832227796\AppData\Local\Temp\tmpmfbr3cimpidlog
[task 2021-12-02T13:40:45.770Z] 13:40:45     INFO - ==> process 6584 launched child process 8736 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.0.984107589\883506245" -parentBuildID 20211202102401 -prefsHandle 1760 -prefMapHandle 1752 -prefsLen 1 -prefMapSize 259509 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 1844 1efb3bedf48 gpu)
[task 2021-12-02T13:40:45.771Z] 13:40:45     INFO - ==> process 6584 launched child process 5244 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.1.134988224\2084906912" -childID 1 -isForBrowser -prefsHandle 2652 -prefMapHandle 2656 -prefsLen 280 -prefMapSize 259509 -jsInitHandle 1280 -jsInitLen 279340 -parentBuildID 20211202102401 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 2696 1efb8875048 tab)
[task 2021-12-02T13:40:45.772Z] 13:40:45     INFO - ==> process 6584 launched child process 6188 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.3.2061643225\1681823554" -childID 2 -isForBrowser -prefsHandle 1240 -prefMapHandle 1236 -prefsLen 5344 -prefMapSize 259509 -jsInitHandle 1280 -jsInitLen 279340 -parentBuildID 20211202102401 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 3508 1efa7c73048 tab)
[task 2021-12-02T13:40:45.773Z] 13:40:45     INFO - ==> process 6584 launched child process 8128 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.5.1174569573\1608144761" -childID 3 -isForBrowser -prefsHandle 3700 -prefMapHandle 3704 -prefsLen 5344 -prefMapSize 259509 -jsInitHandle 1280 -jsInitLen 279340 -parentBuildID 20211202102401 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 3808 1efbc647b48 tab)
[task 2021-12-02T13:40:45.774Z] 13:40:45     INFO - ==> process 6584 launched child process 4964 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.7.1566734638\170333677" -childID 4 -isForBrowser -prefsHandle 4276 -prefMapHandle 4272 -prefsLen 5979 -prefMapSize 259509 -jsInitHandle 1280 -jsInitLen 279340 -parentBuildID 20211202102401 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 4292 1efbcf23048 tab)
[task 2021-12-02T13:40:45.774Z] 13:40:45     INFO - ==> process 6584 launched child process 9076 ("Z:\task_163844832227796\build\application\firefox\firefox.exe" -contentproc --channel="6584.9.26685009\644638488" -parentBuildID 20211202102401 -prefsHandle 3728 -prefMapHandle 5084 -prefsLen 7477 -prefMapSize 259509 -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 5112 1efbbd5e548 rdd)
[task 2021-12-02T13:40:45.775Z] 13:40:45     INFO - ==> process 6584 launched child process 664 ("Z:\task_163844832227796\build\application\firefox\plugin-container.exe" --channel="6584.11.744528978\462243121" "C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\plugins\gmp-fakeopenh264\1.0" -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 5280 1efbe130648 gmplugin)
[task 2021-12-02T13:40:45.776Z] 13:40:45     INFO - ==> process 6584 launched child process 6472 ("Z:\task_163844832227796\build\application\firefox\plugin-container.exe" --channel="6584.14.179401522\1935563242" "C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\plugins\gmp-fakeopenh264\1.0" -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 5904 1efbc519048 gmplugin)
[task 2021-12-02T13:40:45.777Z] 13:40:45     INFO - ==> process 6584 launched child process 3864 ("Z:\task_163844832227796\build\application\firefox\plugin-container.exe" --channel="6584.16.1670513957\1497774301" "C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\plugins\gmp-fakeopenh264\1.0" -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 5568 1efbe128c48 gmplugin)
[task 2021-12-02T13:40:45.778Z] 13:40:45     INFO - ==> process 6584 launched child process 2740 ("Z:\task_163844832227796\build\application\firefox\plugin-container.exe" --channel="6584.18.694652896\1234891380" "C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\plugins\gmp-fakeopenh264\1.0" -appDir "Z:\task_163844832227796\build\application\firefox\browser" - 6584 "\\.\pipe\gecko-crash-server-pipe.6584" 5572 1efa7c64848 gmplugin)
[task 2021-12-02T13:40:45.778Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 8736
[task 2021-12-02T13:40:45.778Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 8128
[task 2021-12-02T13:40:45.779Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 4964
[task 2021-12-02T13:40:45.779Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 6472
[task 2021-12-02T13:40:45.779Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 6188
[task 2021-12-02T13:40:45.780Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 3864
[task 2021-12-02T13:40:45.780Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 9076
[task 2021-12-02T13:40:45.780Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 2740
[task 2021-12-02T13:40:45.781Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 664
[task 2021-12-02T13:40:45.781Z] 13:40:45     INFO - zombiecheck | Checking for orphan process with PID: 5244
[task 2021-12-02T13:40:45.782Z] 13:40:45     INFO - mozcrash Downloading symbols from: https://firefox-ci-tc.services.mozilla.com/api/queue/v1/task/CuDR0jCbS36HwoqPPocA0g/artifacts/public/build/target.crashreporter-symbols.zip
[task 2021-12-02T13:40:48.073Z] 13:40:48     INFO - mozcrash Copy/paste: Z:/task_163844832227796/fetches\minidump_stackwalk\minidump_stackwalk.exe --human C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\minidumps\2884bb93-15a7-47e2-8b68-ebfdc04c4408.dmp C:\Users\task_163844832227796\AppData\Local\Temp\tmpkzyh64n6 --symbols-url=https://symbols.mozilla.org/
[task 2021-12-02T13:40:50.912Z] 13:40:50     INFO - mozcrash Saved minidump as Z:\task_163844832227796\build\blobber_upload_dir\2884bb93-15a7-47e2-8b68-ebfdc04c4408.dmp
[task 2021-12-02T13:40:50.929Z] 13:40:50     INFO - mozcrash Saved app info as Z:\task_163844832227796\build\blobber_upload_dir\2884bb93-15a7-47e2-8b68-ebfdc04c4408.extra
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO - PROCESS-CRASH | Main app process exited normally | application crashed [@ je_free(void*)]
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO - Mozilla crash reason: MOZ_RELEASE_ASSERT((run->mRegionsMask[elm] & (1U << bit)) == 0) (Double-free?)
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO - Crash dump filename: C:\Users\task_163844832227796\AppData\Local\Temp\tmp_erfnf_a.mozrunner\minidumps\2884bb93-15a7-47e2-8b68-ebfdc04c4408.dmp
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO - Operating system: Windows NT
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO -                   10.0.19041 
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO - CPU: amd64
[task 2021-12-02T13:40:51.271Z] 13:40:51     INFO -      family 6 model 85 stepping 7
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -      8 CPUs
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - 
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - Crash reason:  EXCEPTION_BREAKPOINT
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - Crash address: 0x7ffa817fd8f8
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - Process uptime: 341 seconds
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - 
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO - Thread 3 Socket Thread (crashed)
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -  0  mozglue.dll!je_free(void*) [malloc_decls.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 54 + 0x1028]
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rax = 0x00007ffa81885a9b   rdx = 0x0000000000000001
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rcx = 0x0000000000000032   rbx = 0x000001bc842eb001
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rsi = 0x000001bc842eb000   rdi = 0x0000000000040000
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rbp = 0x00000000000dc0b1   rsp = 0x00000074d1c3e610
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -      r8 = 0x0000000000000010    r9 = 0x00007ffa7ab51b28
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     r10 = 0x00000fff4a875dc8   r11 = 0x000001bc842eb360
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     r12 = 0x000001bcf88000a0   r13 = 0x00000074d1c3ef50
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     r14 = 0x00000000000002c1   r15 = 0x000001bc84200000
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rip = 0x00007ffa817fd8f8
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     Found by: given as instruction pointer in context
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -  1  xul.dll!mozilla::MediaEventForwarder<unsigned long long>::~MediaEventForwarder() [MediaEventSource.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 550 + 0x27]
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.274Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e780   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543ae640
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -  2  xul.dll!mozilla::GmpPluginNotifier::~GmpPluginNotifier() [WebrtcVideoCodecFactory.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 29 + 0x7]
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e7b0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543ae5d4
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -  3  xul.dll + 0x391e587
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e7f0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543ae588
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -  4  xul.dll!mozilla::WebrtcVideoConduit::~WebrtcVideoConduit() [VideoConduit.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 411 + 0xb9]
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e830   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.275Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543a22e6
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -  5  xul.dll!mozilla::WebrtcVideoConduit::~WebrtcVideoConduit() [VideoConduit.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 406 + 0x4]
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e890   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543a9fc0
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -  6  xul.dll!mozilla::MediaPipeline::~MediaPipeline() [MediaPipeline.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 268 + 0x77]
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e8d0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543d1ce8
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -  7  xul.dll!mozilla::MediaPipelineReceiveVideo::~MediaPipelineReceiveVideo() [MediaPipeline.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 389 + 0x71]
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e920   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543d6292
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -  8  xul.dll!mozilla::runnable_args_memfn<RefPtr<mozilla::MediaPipeline>,void (mozilla::MediaPipeline::*)()>::~runnable_args_memfn() [runnable_utils.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 110 + 0x35]
[task 2021-12-02T13:40:51.276Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e960   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa543d9866
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -  9  xul.dll!mozilla::Runnable::Release() [nsThreadUtils.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 60 + 0xd]
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e9a0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa5296a7c9
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO - 10  xul.dll!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 1193 + 0x28]
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rsp = 0x00000074d1c3e9d0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa50ab8a79
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO - 11  xul.dll!mozilla::net::nsSocketTransportService::Run() [nsSocketTransportService2.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 1190 + 0x2d]
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     rsp = 0x00000074d1c3ed40   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa52c3299a
[task 2021-12-02T13:40:51.277Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO - 12  xul.dll!nsThread::ProcessNextEvent(bool, bool*) [nsThread.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 1177 + 0xc]
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f1c0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa50ab8749
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO - 13  xul.dll!mozilla::ipc::MessagePumpForNonMainThreads::Run(base::MessagePump::Delegate*) [MessagePump.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 300 + 0x25]
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f530   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa5296b7f0
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO - 14  xul.dll!MessageLoop::RunHandler() [message_loop.cc:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 324 + 0x15]
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f5e0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa514db63f
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO - 15  xul.dll!MessageLoop::Run() [message_loop.cc:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 306 + 0x4]
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f630   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa50ab7c1e
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO - 16  xul.dll!static nsThread::ThreadFunc(void*) [nsThread.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 391 + 0x7]
[task 2021-12-02T13:40:51.278Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f690   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa51336f8b
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO - 17  nss3.dll!_PR_NativeRunThread(void*) [pruthr.c:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 399 + 0xd]
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f840   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa839e3e2c
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO - 18  nss3.dll!pr_root(void*) [w95thred.c:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 139 + 0xc]
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f8c0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa839fb5a1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO - 19  ucrtbase.dll!thread_start<unsigned int (__cdecl*)(void *),1> + 0x41
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f8f0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa89061bb2
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO - 20  kernel32.dll!BaseThreadInitThunk + 0x13
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f920   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa89f77034
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.279Z] 13:40:51     INFO - 21  mozglue.dll!patched_BaseThreadInitThunk(int, void*, void*) [WindowsDllBlocklist.cpp:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 574 + 0x14]
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f950   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa817fbea8
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     Found by: call frame info
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO - 22  ntdll.dll!RtlUserThreadStart + 0x20
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     rbx = 0x000001bc842eb001   rbp = 0x00000000000dc0b1
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     rsp = 0x00000074d1c3f9c0   r12 = 0x000001bcf88000a0
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     r13 = 0x00000074d1c3ef50   r14 = 0x00000000000002c1
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     r15 = 0x000001bc84200000   rip = 0x00007ffa8b142651
[task 2021-12-02T13:40:51.280Z] 13:40:51     INFO -     Found by: call frame info
Group: core-security → media-core-security
Summary: Intermittent Main app process exited normally | application crashed [@ je_free(void*)] → Intermittent Main app process exited normally | application crashed [@ je_free(void*)] from MOZ_RELEASE_ASSERT((run->mRegionsMask[elm] & (1U << bit)) == 0) (Double-free?)

Calling this sec-moderate for now because the problem seems to have been detected by a release assert.

Keywords: sec-moderate
Component: Audio/Video → Audio/Video: GMP
Crash Signature: [@ je_free(void*)] → [@ je_free(void*)] [@ nsTArray_Impl<RefPtr<mozilla::detail::Listener<unsigned long> >, nsTArrayInfallibleAllocator>::ClearAndRetainStorage()]

I found one crash in the wild that looks sort of like this, but it is on ESR-78 so I don't know how relevant it is. bp-1225094e-9264-4910-be59-d332d0211204

Crash Signature: [@ je_free(void*)] [@ nsTArray_Impl<RefPtr<mozilla::detail::Listener<unsigned long> >, nsTArrayInfallibleAllocator>::ClearAndRetainStorage()] → [@ je_free(void*)] [@ nsTArray_Impl<RefPtr<mozilla::detail::Listener<unsigned long> >, nsTArrayInfallibleAllocator>::ClearAndRetainStorage()]

Andreas, does this look like something you're familiar with? I see some GMP in the stack with some WebRTC around it and wonder if this could be related to our recent update.

Flags: needinfo?(apehrson)

GmpPluginNotifier and MediaEventForwarder are both my work from bug 1654112. This is probably not related to GMP as such. By comment 3 I also saw more (older) crashes with ~WebrtcVideoConduit on the crashing stack that all look similar. But they're pre-bug1654112 and may no longer be a problem.

What stands out in the log in comment 0 is WebrtcCallThread #1 which is in [task 2021-12-02T13:40:51.447Z] 13:40:51 INFO - 8 xul.dll!mozilla::detail::ListenerImpl<nsIEventTarget,'lambda at /builds/worker/workspace/obj-build/dist/include/MediaEventSource.h:562:39',unsigned long long>::ApplyWithArgs(unsigned long long&&) [MediaEventSource.h:04a4cc53ce8b6cc25e9cdcba4ccc228adb585762 : 237 + 0x1a]; i.e., in this lambda.

Main thread is crashing in the dtor of an object that created a lambda like that. It seems too big a coincidence that these would be unrelated so I'll assume they're for the same MediaEventForwarder object. We have had some race issues in MediaEventSource that have been fixed, but I'll do an audit for races specific to MediaEventForwarder too.

Component: Audio/Video: GMP → WebRTC: Audio/Video

So the races we fixed in MediaEventSource.h were related to shutdown hangs.

We still have a race, which I believe we exhibit here, if:

  • A MediaEvent was connected with a rawptr used in the event handler, and
  • Disconnection of the listener of said event happens off target, and
  • The lifetime of said rawptr ends off target (likely in the task that did the disconnection), then
  • If the target thread is handling the event and has passed this point when rawptr is destroyed we are in for a race that can lead to a UAF.

It might be time to require disconnection to happen on target. I think that's the easiest fix, but it depends on how many sites are off-target today, and how hard they are to fix.

Alternatively require strong-refs in the event handlers, but as noted in MediaEventSource.h that will affect object lifetimes, and which thread they may be destroyed on. It seems more appropriate to force MediaEventSource users to think and be explicit about their lifetimes.

Assignee: nobody → apehrson
Status: NEW → ASSIGNED
Flags: needinfo?(apehrson)
Priority: -- → P1

I am going to fix just this particular use/race of the MediaEventForwarder in this bug, and do the rest in a followup.

Comment on attachment 9255564 [details]
Bug 1744081 - Move VideoConduit::Shutdown asyncness ownership into VideoConduit. r?ng!, r?bryce!

Security Approval Request

  • How easily could an exploit be constructed based on the patch?: One can easily deduct the threading changes, but finding where the race is seems difficult. There's a a couple of layers of indirection in between.
  • Do comments in the patch, the check-in comment, or tests included in the patch paint a bulls-eye on the security problem?: No
  • Which older supported branches are affected by this flaw?: 96
  • If not all supported branches, which bug introduced the flaw?: Bug 1654112
  • Do you have backports for the affected branches?: No
  • If not, how different, hard to create, and risky will they be?: Very low risk. I don't expect any conflict, but there might be something small.
  • How likely is this patch to cause regressions; how much testing does it need?: The risky part of the patch is that we are changing the thread we run a shutdown step on. But we are not adding asyncness here -- the code is already shutting other things down on that thread. So I think the lifetime properties of the new shutdown sequence are well understood and overall regression risk is low.
Attachment #9255564 - Flags: sec-approval?

Comment on attachment 9255564 [details]
Bug 1744081 - Move VideoConduit::Shutdown asyncness ownership into VideoConduit. r?ng!, r?bryce!

Approved to land and request uplift

Attachment #9255564 - Flags: sec-approval? → sec-approval+
Group: media-core-security → core-security-release
Status: ASSIGNED → RESOLVED
Closed: 3 years ago
Resolution: --- → FIXED
Target Milestone: --- → 97 Branch

The patch landed in nightly and beta is affected.
:pehrsons, is this bug important enough to require an uplift?
If not please set status_beta to wontfix.

For more information, please visit auto_nag documentation.

Flags: needinfo?(apehrson)

Comment on attachment 9255564 [details]
Bug 1744081 - Move VideoConduit::Shutdown asyncness ownership into VideoConduit. r?ng!, r?bryce!

Beta/Release Uplift Approval Request

  • User impact if declined: Potential UAF in webrtc
  • Is this code covered by automated tests?: Yes
  • Has the fix been verified in Nightly?: No
  • Needs manual test from QE?: No
  • If yes, steps to reproduce:
  • List of other uplifts needed: None
  • Risk to taking this patch: Low
  • Why is the change risky/not risky? (and alternatives if risky): This is a race where the fix does change thread usage a bit but theres no significant change in lifetimes which I think is where the biggest risk lies.
  • String changes made/needed:
Flags: needinfo?(apehrson)
Attachment #9255564 - Flags: approval-mozilla-beta?
Regressed by: 1654112
Has Regression Range: --- → yes

Comment on attachment 9255564 [details]
Bug 1744081 - Move VideoConduit::Shutdown asyncness ownership into VideoConduit. r?ng!, r?bryce!

Approved for 96.0b8

Attachment #9255564 - Flags: approval-mozilla-beta? → approval-mozilla-beta+

As part of a security bug pattern analysis, we are requesting your help with a high level analysis of this bug. It is our hope to develop static analysis (or potentially runtime/dynamic analysis) in the future to identify classes of bugs.

Please visit this google form to reply.

Flags: needinfo?(apehrson)
Whiteboard: [sec-survey]
QA Whiteboard: [post-critsmash-triage]
Flags: qe-verify-
Flags: needinfo?(apehrson)
See Also: → 1744589
Group: core-security-release
Duplicate of this bug: 1744589

Copying crash signatures from duplicate bugs.

Crash Signature: [@ je_free(void*)] [@ nsTArray_Impl<RefPtr<mozilla::detail::Listener<unsigned long> >, nsTArrayInfallibleAllocator>::ClearAndRetainStorage()] → [@ je_free(void*)] [@ nsTArray_Impl<RefPtr<mozilla::detail::Listener<unsigned long> >, nsTArrayInfallibleAllocator>::ClearAndRetainStorage()] [@ nsTArray_base<nsTArrayInfallibleAllocator,nsTArray_RelocateUsingMemutils>::ShiftData<nsTArrayInfallibleAlloca…
You need to log in before you can comment on or make changes to this bug.

Attachment

General

Created:
Updated:
Size: